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(54) Method and apparatus for annotation of medical imagery to facilitate patient identification, 
diagnosis and treatment 



(57) A method and apparatus for annotation of med- 
ical imagery to facilitate patient identification, diagnosis, 
and treatment is characterized by an imaging device for 
producing a first signal representative of sensed char- 
acteristics of the individual and a minutiae generator 
which receives the first signal and produces a second 
signal representative of minutiae of the individual, the 
minutiae corresponding to specific branch points of 
blood vessels of the individual A minutiae data genera- 
tor analyzes the characteristics of minutiae and produc- 
es a third signal representative of the characteristics 
which is stored in a minutiae database for each of the 
plurality of known individuals and their medical condi- 
tions. The minutiae and minutiae data may be used to 
annotate medical imagery to facilitate subsequent im- 
age comparison by providing standardized registration 
points and time-varying characteristics. A minutiae 
matcher pairs corresponding second signals and third 
signals from a current patient with those from a data- 
base record, and the paired signals are used to align the 
images and compare them. The minutiae analysis tech- 
niques of the invention can be used to identify medical 
patients, assist in the diagnosis of medical conditions, 
and detect and monitor the use of alcohol and drugs, 
including anesthesia. 




100 



; OtV'Cc 








\XR0PR0CES50R 




MI^JUTt6£ 




OATiBASE 



OUTPUT 

DISPL/K'/PROCESSING 
DEVICE 



FIG. I 



Printed by Joiive, 75001 PARIS (Ffl) 



1 



EP 0 867 830 A2 



2 



Description 

This application is based on provisional applica- 
tions No. 60/022,727 filed July 29, 1996, No. 60/028, 
385 filed Octobe r 15. 1996 and N o. 60/028.387 filed Oc- 5 
tober 15. 1996. 

BACKGROUND OF THE INVENTION 

This invention relates generally to the field of image io 
recognition and processing and specifically to methods 
and systems for identifying diagnosing, and treating 
people based on thermal minutiae within a person's 
body, primarily the face. 

Improved methods for automated access control '5 
and sun/eiilance are vital to ensure the continued secu- 
rity 0! nuclear weapon storage facilities, as well as other 
sensitive or valuable items. Potential threats range from 
terrorist bombings, insider thefts, and industrial espio- 
nage to sabotage by environmental activists. There is 
concern lor increased vigilance in the protection of crit- 
ical strategic assets. 

Current technology being used for access control is 
not sufficient reliable, secure, fast rugged or cost effec- 
tive for routine unattended operations at high -security 25 
locations. The challenge is to develop systems to secure 
facilities and personnel from internal and external 
threats in a cost effective and timely, manner. Replacing 
human guards with automated systems can provide a 
significant cost savings. 3o 

The requirement to positively identify each individ- 
ual seeking access to a facility or to information or serv- 
ices is widespread. Manpower-intensive guard brigades 
are deployed at public functions to protect celebrities, 
and at locations where valuable or important items are 35 
stored. Guards are used to screen entrants based upon 
recognizing either the person or some credential he car- 
ries. Identification credentials such as photo ID badges 
and driver's licenses are widely used for manual identi- 
fication when cashing checks or using credit cards. 
Manual checking of such identification cards may not 
recognize cases where the card is forgery or where the 
person using it is not the rightful owner of the card. To 
assist in solving that problem, more sophisticated iden- 
tifying characteristics may be used on the card, and fea- ^5 
tures may be added to make the card more difficult to 
counterfeit. The use of biometric characteristics such as 
fingerprints, signatures, visual descriptions, or photo- 
graphs is becoming more common. Such information 
can either be readable manually or encoded for reading so 
by an automated system. 

When the identification system is fully automated, 
without a human attendant, biometric sensors at the ac- 
cess location can compare the characteristics of a per- 
son at the location with the stored characteristics of the 55 
person he is claiming to be. When initially issuing per- 
mission for a person to access a biometrically -controlled 
system or location, his biometric characteristics are re- 



corded in the system memory, and also recorded on an 
identification card, for later companson by the system 
controller with his live characteristics. 

BRIEF DESCRIPTION OF THE PRIOR ART 

Current biometric identification systems include use 
of inkless fingerprint systems (called "live scan" units), 
retinal scanners, hand geometry measuring devices, 
voice recognition, handwriting recognition, and facial 
recognition systems which use either visual or infrared 
cameras. Use of fingerprints is generally considered the 
most secure method for positive identification. However, 
when used in an unattended mode, fingerprints can be 
lifted from one location or surface and positioned at an- 
other location. Therefore unattended use of fingerprints 
for identification at locations requiring very- high secu- 
rity is not acceptable. A more common limitation to wide- 
spread use of fingerprints for identification is the require- 
ment for placing one or more clean fingers on a glass 
plate for imaging by the fingerprint recognition system. 
This requires that the hands be free and relatively clean, 
and that the glass plates be maintained intact and clean. 
The plates are vulnerable to vandalism. When used for 
access control at a busy location, there is a time delay 
associated with unloading the hands and positioning the 
fingers properly. Also, users must cooperate with the 
system. In certain scenarios of use, cooperation of the 
subject may be difficult to obtain. Furthermore, many 
persons have a reluctance to being fingerprinted for an 
identity card, since they associate the process with crim- 
inal activities. 

Fingerprints traditionally have been the sole means 
of positive identification admissible as evidence in crim- 
inal trials in the U.S. Fingerprinting of criminals, military 
personnel, persons seeking security clearances, and 
persons applying for sensitive jobs has been performed 
for many years. The FBI established and maintained a 
card file in which each person's fingerprints were printed 
by rolling the fingers first on an inked pad and then on 
the card. Much of the original FBI fingerprint file of rolled 
prints has now been digitized and made available on- 
line for computer access. The process of digitizing the 
historical files, and the continuing task of maintaining 
current fingerprint files, has cost hundreds of millions of 
dollars during the past ten years alone. Aside from the 
labor costs of performing the digitization and managing 
the search tasks through the database, significant R&D 
has been performed to develop specialized software for 
comparing unknown fingerprints against the database 
within a reasonable period of time, and specialized hard- 
ware has been developed to provide rapid response. 

Inkless techniques are now generally used to pro- 
duce a "tenprint" card which substitutes for the former 
rolled print card. Common inkless techniques utilize po- 
larized light to illuminate the fingers, and light sensors 
to image the light reflected and refracted from the ridges. 
The resulting image can be more consistent and higher 
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quality than the rolled prints, since inconsistencies in the 
annount of ink applied and in the pressure used to trans- 
ler the print to paper are not a factor. 

Autonnated fingerprint matching techniques have 
been developed which rapidly classify an unknown print 
and then search through the portion of the database as- 
sociated with that class looking tor a match. Unknown 
prints may be from a "tenprint" card, or may be latent 
prints which have been lifted from a crime scene. A la- 
tent print may include a sizable area of one or more fin- 
gers, such as on a water glass, or it may include only a 
portion of one or more fingers, such as on a telephone 
keypad. Latent prints may be found on top of other latent 
prints, such as when several people have used the 
same telephone. 

fVlatching techniques often extract minutiae points 
from the prints, and then compare the sets of minutiae 
rather than compare entire prints. Various classifica- 
tions of minutiae types have been proposed by different 
companies and authorities. An example is given from 
the Costello U.S. Patent No. 4,947,443. Six types 
ofcharacteristic features" are presented in this patent, 
each one relating to a type of minutia. This fingerprint 
matching technique references the type, orientation^ 
and location of each characteristic and each and every 
other characteristic. Using this approach, on the order 
of 80 to 1 50 minutia points are identified in each finger- 
print. Other fingerprint minutiae extraction and matching 
patents produce essentially the same number of minu- 
tiae, with difference in what features of the set of minu- 
tiae are considered in attempted matching and in how 
the matching is performed. In U.S. courts, evidentiary 
rules have traditionally required that 16 or more minutia 
points be found to correspond between two prints in or- 
der for ihem to be considered to be from the same per- 
son. The determination of likely matching prints is gen- 
erally assisted or performed entirely by a computer sys- 
tem; however, the final decree of a match is made by a 
fingerprint expert, who reviews the computer system re- 
sults. 

Matches between different prints taken from the 
same finger are never perfect, since the fingers are de- 
formable. three-dimensional, connected and jointed 
structures which leave two-dimensional prints on sur- 
faces they encounter through pressure. The exact an- 
gles between the fingers and the surfaces, the amount 
and direction of pressure, and the effect of movement 
between the fingers and the surfaces all cause varia- 
tions in the exact prints produced. Even when prints are 
produced by a live scan technique, variation in the light- 
ing, hand position, oil or dust on the fingers, use of lo- 
tions, and scratches or paper cuts will produce minor 
variations in the prints produced. 

Therefore, the exact number, position, and charac- 
teristics of minutiae extracted from two prints may be 
different even though they are produced by the same 
finger. The challenge for an automated fingerprint iden- 
tification system is to recognize allowable minor varia- 



tions in actual matching prints while not allowing varia- 
tions so wide that mismatches occur. Several AFIS 
products are now commercially offered which provide 
acceptable accuracy Local and regional police forces 

5 may use smaller databases which contajn onlyjhe prints 
of persons historically associated with their areas, rather 
than relying on federal resources to search the entire 
nationwide FBI fiies. Smaller scale fingerprint system, 
such as those associated with a system which controls 

10 access to an office building, may use the same minutiae 
matching techniques. 

With rolled and live-scan prints, the orientation 
ofeach print, and the finger to which it corresponds is 
know. Also, quality checks can be built into the process 
such that repeat prints may be taken to insure quality 
when needed. In the case of latents, however, the anal- 
ysis is done after the fact. It is not known which finger 
left the print, and the orientation of the finger may be in 
doubt when only a partial print is found. Therefore. 

20 matching of latents is much more difficult than matching 
of rolled or live scan prints. 

Various minutiae extraction algorithms are used in 
current fingerprint identification systems, some of which 
merely utilize the location of the minutia points and oth- 

25 ers of which utilize also additional information about the 
type of minutia each point represents. For example, sim- 
ple graph matching techniques can be used to compare 
the follow-the-dots vectors generated by connecting the 
minutia points in order forced by considering intersec- 

30 tions with a spiral from the centerpoint of the fingerprint. 
Alternately, the ridge angle at each minutia point can be 
considered and matched along with the coordinates, in 
a best-fit attempt to match each unknown print to each 
known print. A measure of goodness of fit can then be 

35 computed and used to rank other possible matches. 

U.S. patent No. 4,525.859 to Bowies teaches a pat- 
tern recognition system which detects line bifurcations 
and line endings, denoted minutiae, in a pattern of lines 
such as are found in a fingerprint. According to this ref- 
erence, the FBI uses an automatic fingerprint identifica- 
tion system entitled "FINDER" which uses an optical 
scan reader. The information is then enhanced to elim- 
inate grays and fill in gaps in the ridges. A 16x16 incre- 
ment square window scans the fingerprint, an increment 

^5 being a tenth of a millimeter. Thus, a window advances 
through the fingerprint in increments of tenth of millim- 
eter and looks for ridges which enter the window but do 
not exit it. When such a ridge is identified, its coordinate 
location is stored and the ridge is analyzed to establish 

50 an angle, theta, of the ridge at the termination. The data 
are then re-scanned to look for terminations of valleys, 
which are ridge bifurcations. The additional coordinates 
and angles of each of the inverted ending points also 
are stored. 

55 In latent prints, the distances between ridges of a 
fingerprint average 0.4 millimeters but can vary by a fac- 
tor of 2 for any individual finger depending on skin dis- 
placement when the finger contacts the hard surface 
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normally encountered in establishing a print. 

A known algorithm of the National Institute for 
Standard Technology can be used to connpare a previ- 
ously stored electronic image of minutiae coordinate lo- 
cations with the minutiae locations identified and stored s 
by the computer. 

U.S. Patent No. 5.040,224 to Hara discloses a fin- 
gerprint processing system capable of detecting a core 
of a fingerprint image by statistically processing param- 
eters. Hara's invention provides a system to determine 
a core in the fingerprint image and/or to detect directions 
and curvatures of ridges of the fingerprint image prior to 
detection of the position of the core. This reference de- 
fines minutiae as abrupt endings, bifurcations, and 
branches. ^5 

U. S Patent No 4. 790,564 to Larcher teaches a proc- 
ess and apparatus for matching fingerprints based upon 
comparing the minutiae of each print in a database with 
precomputed vector images of search minutiae in a 
search print to be identified, comparing position and an- 20 
gle, a result of such companson being a to be matching 
score indicating the probability- of a match between the 
angle of a file print minutiae and the angle of precom- 
puted veclor images of the search minutiae. Over or un- 
der-inking of a rolled print can change the apparent type ^5 
of minutiae associated with a particular point from one 
printing to the next. However, not all corresponding mi- 
nutiae will appear to change type in the two prints. 
Therefore, matching for type as well as for x and y co- 
ordinates provides a stricter match requirement and re- 
suits in better system accuracy Larcher assigns higher 
values to minutiae which match in x,y and type. 

As Larcher points out. there are advantages to 
matching minutiae rather than the entire image of the 
fingerprint in itself. An elementary matching operation 35 
comprises the comparison of two sets of minutiae, i.e., 
two sets of points, each point having three coordinates 
X, y and a. An elementary matcher attempts to super- 
impose the two sets of points, in order to count the 
number of minutiae which are common to the two fin- 40 
gerprints. 

Numerous other schemes for matching fingerprints 
are known. For example matchers referred to in Weg- 
stein, Technical Note 538 of the National Bureau of 
Standards (1970), as M19, f\^27, and M32, determine ^5 
whether two fingerprints come from the same finger by 
computing the density of clusters of points in Dx-Dy 
space, where Dx and Dy are the respective differences 
in X and y coordinates for the minutiae of two finger- 
prints. Experimental results referred to in this reference so 
indicate that in Dx-Dy space points tend to be located 
at random when coming from different fingerprints, 
whereas points tend to form a cluster when coming from 
fingerprints from the same finger. 

In the M19 matcher, the assumption is made that 55 
the transformation needed to superimpose the two sets 
of minutiae points is a translation only. The M27 matcher 
is an Ml 9 matcher with a new scoring function, intended 



to take into account greater translation displacements. 
The M32 matcher takes into account small rotations be- 
tween two fingerprints in the following way: first an M27 
matcher comparison is made between the two finger- 
prints; then, one of the two prints is rotated through 'V" 
degrees from its original position and a new M27 com- 
parison is made. All together an M32 matcher operation 
consists of seven M27 comparisons, corresponding to 
the following values for the angle V, i e., V= - 15. -10, 
-5, 0, +5, +10, +15 degrees. 

Minutiae may be extracted manually or automatical- 
ly Automatic systems generally require better quality 
imagery. The matcher engine must allow for some de- 
gree of inaccuracy or variability with respect to each of 
the encoded coordinates due to human operator bias or 
precision limitations of automated feature extraction 
processes. 

Larcher disclosed the use and comparison of type 
of minutiae, since there is a greater match accuracy 
when ridge endings are compared to ridge endings, and 
bifurcations to bifurcations, as opposed to comparing 
one ridge ending to one bifurcation. 

Other known approaches compare two sets of im- 
age features points to determine if they are from two 
similar objects as disclosed for example in Sclaroff and 
Pentland, MIT Media Laboratory, Perceptual Computing 
Technical Report #304. This reference suggests that 
first a body-centered coordinate frame be determined 
for each object, and then an attempt be made to match 
up the feature points. 

Many methods of finding a body-centered frame 
have been suggested, including moment of inertia meth- 
ods, symmetry finders, and polar Fourier descriptors. 
These methods generally suffer from three difficulties: 
sampling error, parameterization error, and non-unique- 
ness. The technique used in Sclaroff and Pentman dis- 
closure has the limitation that it cannot reliabiy match 
largely occluded or partial objects. 

Known techniques associated with fingerprint minu- 
tiae extraction and matching can be summarized as fol- 
lows: 

First, an unknown finger is scanned optically; 

Second, the image is divided into pixels, where the 
size of the pixel relates to the quality of the result 
desired; 

Third, certain pixels are selected as minutiae points; 

Fourth, each minutia is assigned a vector having 
magnitude and directional information in relation to 
the surrounding characteristics of the fingerprint; 
Typically for each fingerprint, there would be a sub- 
stantial number of minutia vectors characterizing its 
image; 

Fifth, the set of minutia vectors of the unknown print 
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are compared by computer to the set of vectors of 
known prints; and 

Sixth, the comparison results are used to select po- 
tential matches and provide a goodness of fit indi- s 
cation between the unknown and known prints. 

Numerous approaches to recognition using visible 
light imaging of faces have been proposed. Many of 
them apply standard pattern matching techniques: oth- io 
ers involve definition of face metrics. 

U.S. patent No. 4,975,969 to Tal discloses a method 
and apparatus for uniquely identifying individuals by 
measurement of particular physical characteristics 
viewable by the naked eye or by imaging in the visible '5 
spectrum. This reference defined facial parameters 
which are the distances between identifiable parame- 
ters on the human face, and/or ratios of the facial pa- 
rameters, and teaches that they can be used to identify 
an individual since the set of parameters for each indi- 20 
vidual is unique. 

Tal's approach utilizes visible features on the face, 
and therefore cannot be relied upon to distinguish be- 
tween faces having similar visual features, for example 
as would be the case with identical twins. In addition, ^5 
the "rubber sheeting" effect caused by changes in facial 
expression, ihe aging effects which cause lengthening 
of the nose, thinning of the lips, wrinkles, and deepening 
of the creases on the sides of the nose, all cause chang- 
es in the parameters and in ratios relied on in this meth- 30 
od. Furthermore, the parameters and ratios of any par- 
ticular person's face may be measured by anyone taking 
a photograph, and thereby used to select or disguise 
another person to appear to be that person. Therefore, 
the security provided by such a technique may not be 35 
adequate for unattended or highly sensitive locations. 

Still another known scheme utilizes eigenanalysis 
of visual face images to develop a set of characteristic 
features. Pentland, View-Based and Modular Ei- 
genspaces for Face Recognition, MIT Media Laboratory 40 
Perceptual Computing Section, Technical Report No. 
245. Faces are then described in terms of weighting on 
those features. The approach claims to accommodate 
head position changes and the wearing of glasses, as 
well as changes in facial expressions. This disclosure 
leaches that pre-processing for registration is essential 
to eigenvector recognition systems. The processing re- 
quired to establish the eigenvector set is extensive, es- 
pecially for large databases. Addition of new faces to 
the database requires the re-running of the eigenanal- so 
ysis. Accordingly, use of eigenanalysis may not be ap- 
propriate for use in a general face identification system 
such as would be analogous to the FBI's and AFIS fin- 
gerprint system. 

Visible metrics typically require ground truth dis- ss 
tance measurements unless they rely strictly upon ratios 
of measurements. Thus, such systems can be fooled by 
intentional disguises, and they are subject to variations 



caused by facial expressions, makeup, sunburns, shad- 
ows and similar unintentional disguises. Detecting the 
wearing of disguises and distinguishing between iden- 
tical twins may be done from visible imagery if sufficient 
resolution and controlled lighting is available.^ However.^ 
that significantly increases the computational complex- 
ity of the identification task, and makes the recognition 
accuracy vulnerable to unintentional normal variations. 

From the standpoint of evidentiary use, it might also 
be argued that the application of eigenanalysis to a very 
large database of faces, such as all mug shots in the 
FBI files, would be considered so esoteric by the public 
at large that automated matches based upon its use will 
not readily be acceptable to a jury as convincing evi- 
dence of identity. By comparison, techniques based on 
minutiae matching technique, such as are used with fin- 
gerprint identification, would be expected to find a more 
understanding reception by the law enforcement com- 
munity, and to be more acceptable for evidentiary pur- 
poses within a reasonable number of years after their 
introduction. 

One known scheme using facial thermograms for 
identification is described in the Prokoski et al U.S. Pat- 
ent No. 5,163,094 which discloses defining "elemental 
shapes" in the surface thermal image produced by the 
underlying vascular structure of blood vessels beneath 
the skin. Depending on the environment of use, thermal 
facial identification may provide greater security over 
identification from visual images and may therefore be 
considered preferable. It is extremely difficult, if not im- 
possible, to counterfeit or forge one face to look like an- 
other in infrared, whereas it is often possible to disguise 
one person to look like another in visible fight. However, 
the use of elemental shapes is found in practice to be 
vulnerable to such variables as head rotation and tilt, 
ambient and physiological temperature changes, varia- 
tions in imaging and processing systems, and distor- 
tions or obstructions in a facial image (e.g., due to eye- 
glasses). 

Eigenanalysis of the elemental shapes of a thermal 
facial image has also been used for recognition. In one 
approach, several sets of elemental shapes are pro- 
duced for each image by imposing different thermal 
banding constraints. The totality of shapes are then an- 
alyzed with respect to a library of facial thermal images. 
Eigenshape analysis is used to compare the character- 
istics of shapes in each person's images. Eleven char- 
acteristics of each shape are considered, including: pe- 
rimeter, area, centroid x and y locations, minimum and 
maximum chord length through the centroid, standard 
deviation of that length, minimum and maximum chord 
length between perimeter points, standard deviation of 
that length, and area/perimeter. 

Each person's image is then characterized by a set 
of 11 -coefficient vectors. The difference in eigenspace 
between any two images is calculated to yield a meas- 
urement to which a threshold was applied to make 
a*match/no match" decision. In practice, such a system 
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yields a useful method and apparatus for some applica- 
tions. However the calculation techniques for such a 
system are computationally intensive and require addi- 
tional computational analysis of the entire database 
when new images are added. As with others of the prior s 
known techniques, recognition is seriously impacted by 
edge effects due to head rotation and tilt, and by loss of 
definition in very cold or very hot faces. 

None of the known techniques for facial analysis is 
believed to be sufficiently robust and computationally io 
straightforward to allow practical application of such a 
scheme for highly sensitive unattended security appli- 
cations. 

Therefore, the need remains for a system and meth- 
od that can be used to reliably recognize and verify the '5 
identity of an imaged person without manual assistance 
and without cooperation from the person being identi- 
fied. 

SUMMARY OF THE INVENTION 20 

In accordance with the present invention, a system 
for recognizing faces comprises a thermal imaging de- 
vice, a minutiae generator, a minutiae data generator, 
and a minutiae matcher The thermal imaging device 25 
produces a signal representative of the thermal charac- 
teristics of a new face. The minutiae generator is con- 
nected to the thermal imaging device and produces a 
signal representative of thermal facial minutiae of the 
new face. The minutiae data generator stores minutiae 
data corresponding to known faces. The minutiae 
matcher is connected to the minutiae generator and the 
minutiae data generator and compares minutiae of the 
new face and of the known faces, producing a signal 
representative of a match between the new face and 3S 
one of the old faces 

In another aspect of the invention, a method of rec- 
ognizing faces senses thermal characteristics of known 
faces, identifies minutiae of the known faces, senses 
thermal characteristics of a new face, identifies minutia 40 
of the new face, determines a distance metric from each 
of the known faces to the new face, and determines a 
match between the new face and one of the old faces 
based on the distance metrics. 

In still another aspect of the invention, faces are 
classified according to thermal minutiae, and facial mi- 
nutiae data are encoded as a number of bits by overlay- 
ing a grid of cells on a thermal representation of face, 
setting a bit to a first state if any minutiae are located 
within the cell corresponding to that bit, and setting the 50 
bit to a second state if none ofthe minutiae are located 
within the cell corresponding to that bit. 

In yet further aspects of the invention, other imaging 
modalities are used, and other body parts or objects are 
used, for minutiae-based recognition. Techniques for ss 
identifying medical patients, diagnosing medical condi- 
tions, identifying drug and alcohol users, and assisting 
with the positioning of surgical instruments are also 



achieved with the present invention. 

BRIEF DESCRIPTION OF THE FIGURES 

Other objects and advantages of the invention will 
become apparent from a study of the following specifi- 
cation, when viewed in the tight of the accompanying 
drawing, in which: 

Fig. 1 is a block diagram of a system for recognizing 
faces according to the present invention; 

Figs. 2a and 2b are front and side views, respec- 
tively, of the vascular system of the human head, 
with the location of thermal minutiae being indicated 
in Fig. 2b; 

Fig. 2c is a schematic view of the vascular system 
of the human brain; 

Fig. 3 is a front view of the human vascular system 
illustrating the location of thermal minutiae therein; 

Fig. 4 illustrates a contour plot derived from a facial 
thermogram and identified facial features in accord- 
ance with the invention; 

Fig. 5 illustrates a contour plot derived from a facial 
thermogram and minutia points in accordance with 
the invention; 

Fig. 6 illustrates a grid of cells overlaid on a contour 
plot derived from a facial thermogram in accord- 
ance with the invention; 

Fig. 7 is facial thermogram taken from a distance of 
approximately fifteen feet in accordance with the in- 
vention; 

Fig. 8 is a block diagram of the apparatus for anno- 
tating an image of the human body according to the 
invention; 

Fig. 9 is a block diagram of a modified apparatus of 
Fig. 8 for identifying reference minutiae in an anno- 
tated image; 

Figs. 10a. 10b, and 10c illustrate facial minutiae su- 
perimposed on three different thermal images, re- 
spectively of the same face; 

Fig. 11 is a block diagram of an apparatus for an- 
notating a medical image of the human body ac- 
cording to the invention; 

Fig. 12 is a thermogram of the upper chest area of 
an individual taken from a distance of approximately 
fifteen feet in accordance with the invention; 
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Fig 1 3 illustrates the corresponding anatomy for the 
thermogram of Fig. 12; 

Fig. 14 is a block diagram representing apparatus 
for maintaining the position of a surgical instrument s 
relative to a surgical site according to the invention; 

Fig. 15 IS a facial thermogram of an alcohol-free in- 
dividual; 

10 

Fig. 16 is a facial thermogram of the individual of 
Fig. 15 under the influence of alcohol; 

Fig. 17 is a graph representing the thermal signa- 
tures of selected minutia points of an individual prior '5 
and subsequent to use of alcohol; 

Fig. 18 is a block diagram of the apparatus for de- 
tection of alcohol and drug use by an individual ac- 
cording to the invention; 20 

Fig. 19 is a graph representing the results of a sta- 
tistical analysis of drug users in a random popula- 
tion; and 

25 

Fig. 20 is a block diagram of the apparatus for de- 
tection of alcohol and drug use in a random popu- 
lation. 

DETAILED DESCRIPTION 30 
Facial i\/1inutiae Extraction 

In Fig. 1 , there is shown a system 100 for personal 
identification in accordance with the present invention. 35 
System 100 includes seven major subsystems: a ther- 
mal imaging device 102, minutiae generator 104, a mi- 
nutiae data generator 105, minutiae matcher 106, minu- 
tiae database 111, microprocessor 1 1 0, and output dis- 
play/processing device 1 1 4. In a preferred embodiment, 
minutiae generator 104, minutiae data generator 105, 
and minutiae matcher 106 are all implemented by pro- 
gram instructions stored in a program memory 108, and 
program memory 108, microprocessor 110, and minuti- 
ae database 111 are implemented by a programmed ^5 
conventional computer 112. 

In operation, thermal imaging device 102 obtains a 
thermal image of the face of person 1 01 . A digital signal 
representative of the thermal image is provided as input 
to minutiae generator 104, which generates signals rep- so 
resentative of thermal facial minutiae points for 101. 
These minutiae points have a number of characteristics 
including a specific location within the person or relative 
to other minutia, the apparent temperature at a given 
time, the temperature signature over a period of time, 55 
whether the minutia corresponds to a vein or artery, the 
width of the blood vessel, and the vector direction of 
branching blood vessels from the minutia. These and 



other characteristics are sensed and data relative there- 
to are generated by a minutia data generator 105. This 
data is stored in the minutiae database 111. The minu- 
tiae matcher 106 compares minutiae data for known in- 
dividuals which has been stored in the data base _with. 
that for unknown individuals currently being imaged by 
the imaging device 102. If a match is detected, a corre- 
sponding signal is sent to the output display/processing 
device 114. 

In a preferred embodiment, output display/process- 
ing device 1 1 4 comprises circuitry to permit or deny ac- 
cess to a secured facility depending on the results of the 
matching performed by minutiae matcher 106. In one 
embodiment, access is permitted if the person 101 is 
recognized as one of a group of authorized personnel. 
In a second embodiment, access is denied if the person 
1 01 is recognized as one of a group of unauthorized per- 
sonnel. In yet another embodiment, access is denied if 
the person 101 is not recognized by system 100. 

System 1 00 thus considers hidden micro parameter 
which lie below the skin surface, and which cannot be 
easily forged, if at all. The large number of such micro 
parameters considered renders it essentially impossible 
to search for a person to match another person's set of 
micro parameters. Furthermore, the particular infrared 
band used for imaging by thermal imaging device 102 
may be kept secret, or multiple bands may be used, 
which further increases the difficulty involved in compro- 
mising system 1 00. The underlying features detected by 
system 100 are essentially "hardwired" into the face at 
birth and remain relatively unaffected byaging, thus pro- 
viding for less inherent variability than found in known 
recognition systems. Although thermal facial minutiae 
have some aspects related to, and extractable from, el- 
emental shapes and may be tagged to reflect the ele- 
mental shape parameters (such as by tagging with frac- 
tal dimensions), minutiae extraction does not require 
production or consideration of elemental shapes. Fur- 
thermore, the comparison of thermal facial minutiae is 
computationally straightformard and introduces signifi- 
cantly less processing overhead than the known ap- 
proaches used for template or shape comparisons. 

Thermal imaging device 1 02 may be any device that 
produces a signal representative of the thermal charac- 
teristics of the face of person 1 01 . in a preferred embod- 
iment, a conventional digital video camera sensitive to 
thermal energy is used for the thermal imaging device 
102. As described herein, it is found that tractable im- 
agery for facial identification may be derived from pas- 
sively obtained infrared images of facial heat emana- 
tions which can be detected by commercially available 
thermal imaging devices sensitive in the 3 to 12 micron 
wavelength band. Unlike fingerprints that are character- 
ized by a limited range of intensity values corresponding 
to three dimensional ridges which are essentially con- 
centric rings about a single center, plus anomalous arch- 
es, line endings, and bifurcations, facial thermograms 
are generally characterized by continuously varying 
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wide distribution of temperatures, including multiple 
maxima and minima values. Where the skin surface is 
unbroken, there is gradual variation of temperatures 
from the hot areas on either side of the nose to the rel- 
at ively cool a reas of th e ea rs and chee ks. The eyes ap- 5 
pear to be cooler than the rest of the face. The nostrils 
and mouth, and surrounding areas, will look warm or 
cool depending upon whether the subject is inhaling or 
exhaling through them. Discontinuities in the skin sur- 
face temperature may be evident where scars, moles, io 
burns, and areas of infection are found. 

In some applications, thermal imaging device 102 
may be adapted for attended operation using coopera- 
tive persons 101 and a human supervisor, as with iden- 
tification systems based on roiled fingerprints. In these is 
applications, the supervisor can ensure that person 101 
is properly positioned, and can adjust gain, focus, and 
other parameter of thermal imaging device 102 to opti- 
mize the quality of the thermal image produced by ther- 
mal imaging device 102. In other applications, thermal 
imaging device 102 is adapted for unattended, stand- 
alone operation, for instance with live scans used for ac- 
cess control to a remote secure facility. System 1 00 can 
further be configured based on an expectation that per- 
son 1 00 will be either cooperative (e.g. , moving to a spe- 25 
cific requested location for optimal imaging) or uncoop- 
erative (e.g., a mere passer-by). In environments where 
uncooperative persons are expected, identification wili 
be facilitated by collecting the maximum possible 
amount of data, for instance by using multiple thermal 30 
imaging devices 102 and fast frame (i.e., samplings 
rates. Additional related data, referred to herein as 
"ground truth" data, may be collected as well to provide 
information on factors such as ambient temperature, ab- 
solute size of the imaged face, or the distance of the 3S 
imaged face from thermal imaging device 102. 

Any portion of the body can be utilized for identifi- 
cation, but the face is preferred due to its typical acces- 
sibility for imaging. In Fig. 2a there is represented the 
vascular system for a human face and in Fig. 2b there 40 
are shown minutiae points 1 50 for the face of Fig. 2a. In 
Fig 3, selected minutiae points 1 50 throughout a human 
body are shown. 

Since parts of the face may be blocked by glasses, 
facial hair, or orientation to thermal imaging device 102, ^5 
system 100 provides for identification based on partial 
faces. A sufficient number of minutiae may be obtaina- 
ble from portions of the face not blocked by glasses, fa- 
cial hair, or other concealment to permit matching. Al- 
ternately, if fewer than a minimum number of minutiae so 
specified for a particular scenario are extracted by sys- 
tem 100 for a particular person 101 in an unattended 
setting, that person 101 may be considered by system 
100 to be potentially disguised, and output/display 
processing device 1 1 4 may cause an alarm to be gen- ss 
erated to alert guard personnel to that possibility. 

Various perturbations, such as facial expression 
changes, can distort the relative locations of minutiae 



points. This is analogous to the deformations that occur 
in fingerprints due to movement or pressure between 
the fingers and the print surface. As described below 
minutiae matcher 106 allows for some variations in the 
position and characteristics of the minutiae, as well as- 
in the subset of minutiae which are seen due to the field 
of view of thermal imaging device 102 and to possible 
obstruction of certain areas of the face in the image. 

As set forth in greater detail herein, in one embod- 
iment the minutiae database 112 is partitioned by clas- 
sifying data corresponding to faces based on minutiae- 
related characteristics as generated by the minutiae da- 
ta generator 105. In alternative embodiments, other 
characteristics may be used for such classification. 
Such classification is found to reduce search require- 
ments in connection with the operation of database 112 
and minutiae matcher 106. 

Minutiae Generator 104 

In a preferred embodiment, minutiae generator 104 
performs seven major functions: designation of faces 
axes, testing of face axes validity; normalization; pro- 
duction of thermal contour lines; establishment of 
threshold radius of curvature; selection of minutiae, and 
assignment of characteristics to minutiae. Each of these 
functions is described in greater detail below. 

1. Designation of Face Axes 

Referring now also to Fig. 4, there is shown a facial 
thermogram 200 as produced by thermal imaging de- 
vice 102. The thermogram is in the form of a contour 
plot derived as set forth below In a preferred embodi- 
ment, thermogram 200 produced by thermal imaging 
device 102 is represented by digital signals, but if an 
analog thermal imaging device 102 is used, minutiae 
generator 104 can include conventional analog-to digital 
conversion circuitry to obtain facial thermogram 200 as 
a digital signal representation of the face of person 101. 

Once the facia! thermogram 200 is produced, minu- 
tiae generator 104 locates a number of facial features 
on thermogram 200, either with manual assistance or 
automatically by using conventional techniques and 
structures as described herein: left and right canthi 201 , 
202, left and right nostrils 203, 204, and mouth area 205. 

In a preferred embodiment, minutiae generator 104 
locates the left canthi area and the right canthi area and 
determines the centroid for each area. The location of 
the centroid is essentially independent of the grey scale 
allocation of the analog thermal image produced by the 
camera 102. The centroids are referred to as left and 
right canthi 201 . 202 herein. 

Minutiae generator 1 04 also determines an eye line 
206 between left and right canthi centroids 201. 202. 
This being done, minutiae generator determines a 
mouth line 205 parallel to eye tine 206 and passing 
through the centroid of mouth area 205. Minutiae gen- 
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erator 104 next determines a vertical central line 208 
perpendicular to eye line 206 and mouth area 205 and 
intersecting eye line 206 midway between left and right 
canthi centroids 201, 202. Minutiae generator 104 then 
determines a face center point 209 on central line 208 
midway between the points of intersection of vertical 
central line 208 with eye line 206 and mouth 205. Minu- 
tiae generator 1 04 further determines a horizontal cent- 
er line 210 perpendicular to the vertical central line 208 
and passing through face center point 209. Vertical cen- 
tral line 208 and horizontal central line 210 are desig- 
nated as face axes. Numerous other features may be 
used to define face axes, but in general it is preferable 
to define face axes based on areas of the face that are 
not greatly deformable. 

Other techniques may be used for location of the 
face center point 209 in those cases where the preferred 
use of facial symmetry and recognizable thermal fea- 
tures does not suffice. For example, other techniques 
may be called for with respect to facial images in which 
an eye patch is worn, eyeglasses are not symmetrical, 
only a partial face is imaged, the lower face is covered 
or the thermal pattern of the face is unusually distorted. 
The face center point 209 may in fact be outside of the 
boundaries of the facial image, for instance where only 
a partial facia! image is obtained due to the face being 
partially blocked by another face or some other object. 
If the person 101 is wearing glasses, the pattern of the 
glasses, which typically block the infrared emissions 
from the face and thereby produce an extended cold ar- 
ea with sharp thermal discontinuity, can be used to de- 
termine approximate face axes. Additional techniques 
include manual location of the face center point 209 and 
preprocessing using known techniques to locate the ap- 
proximate area of the face center point 209. As de- 
scribed below the face axes may be tested for validity 
to determine whether the image requires any such spe- 
cial treatment. 

II. Testing the Validity of Face Axes 

Since the known techniques for identifying left and 
right canthi centroids 201, 202, left and right nostrils 
203. 204, and mouth area 205 are subject to artifacts 
and other sources of error, and since some images of 
faces are significantly asymmetric or have features that 
are entirely missing {e.g., due to person 101 wearing an 
eye patch or having a disfigured face), minutiae due to 
person 101 wearing an eye patch or having a disfigured 
face), minutiae generator 104 performs checks to help 
spot instances where these points may have been in- 
correctly located or where unusual facial images are en- 
countered. First, a check is made to ensure that vertical 
central line 208 and mouth line 207 intersect within 
mouth area 205. Next, a check is made to ensure that 
vertical centra! line 208 intersects a line connecting left 
and right nostrils 203, 204 at point between left nostril 
203 and right nostril 204. If either of these conditions is 



not met the face is considered to be a special case call- 
ing for manual intervention to determine the best ap- 
proximation for face axes. 

5 III. Normalization 

In practice, it is found that preprocessing through 
normalization of image size provides advantages in later 
recognition. Accordingly, minutiae generator 104 uses 

^0 the distances between left and right canthi centroids 201 
and 202 and the distance from face center 209 to eye 
line 206 to compare the size of facial thermogram 200 
with a standard image size. In a preferred embodiment, 
linear correction in the vertical and horizontal dimen- 

^5 sions is used to normalize the size of facial thermogram 
200 to match the standard, but other normalization mod- 
els could be used as well. 

IV. Production of Thermal Contour Lines 

20 

As provided by thermal imaging device 102, facial 
thermogram 200 consists only of an ordered list of ther- 
mal values corresponding to each small portion of the 
imaged face. Minutiae generator 104 employs the loi- 
2S lowing procedure to produce thermal contour lines for 
facial thermogram 200: 

a For a digitized image having N bits ol resolution, 
or 2^ bands of thermal values, determine thermal 
30 contour lines having a particular "current" one of the 
2Nvalues. 

b. Produce minutiae in accordance with the steps 
below for the contour lines of the current value. 

35 

c. Repeat a and b above, each time using new one 
of the 2^^ values for the "current" value, until the de- 
sired number of minutiae have been extracted of all 
of the possible values have been processed. 

40 

6. If the desired number of minutiae have not been 
extracted, repeat the process beginning with 2^''^ 
bands of values, and reduce the number of bands 
by 1 with each iteration, skipping those that are 
45 powers of 2, until the desired number of minutiae 
have been extracted or until no further reduction in 
bands can be achieved. 

Various other techniques for generating contour 
50 lines may also be used, with the goal being obtaining a 
sufficiently large number of minutiae for unique recog- 
nition, without producing too many spurious minutiae. 
Spurious minutiae increase processing overhead with- 
out benefitting recognition. The number of thermal 
55 bands that will produce an appropriate number of minu- 
tiae is readily determined by trial and error for any par- 
ticular application of system 100. 
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V. Establishment of Maximum Radius of Curvature 

In a preferred embodiment, points on a thermal con- 
tour are considered minutiae if they form inflection 
points for the contour. However, to avoid artifacts.result-- 5 
ing in too many minutiae being selected , only inflection 
points for curves below a threshold radius will be con- 
sidered minutiae. Therefore, minutiae generator 104 se- 
lects a maximum radius of curvature to be used in de- 
termining minutiae, based on characteristics of system io 
100 such as the resolution of thermal imaging device 
102, the lens used, the quality of the recording and 
processing system, the desired number of minutiae to 
be extracted, the desired sensitivity and vulnerability of 
the system to minor variations in thermal image, the ac- ^5 
curacy of the three dimensional model for registration of 
the face image, and the magnitude of systematic and 
random errors. 

VI. Selection of Minutiae 20 

Since the face thermal surface can be distorted 
through changes in expression, activities such as eating 
and talking, tight hats and other clothing, sinus inflam- 
mation, and weight gain and loss, the minutiae points to 25 
be extracted must remain fairly constant in spite of such 
changes, or must be able to be filtered through those 
changes. Section of minutiae as described herein pro- 
vide minutiae weil-suited to such factors. 

Minutiae generator 104 selects minutiae from the 30 
facial thermogram 20O after preprocessing as described 
above by first positioning a circle of radius R on a ther- 
mal contour such that the contour intersects the circle, 
crossing it at two points and dividing it with equal area 
in each half. Next, this circle is moved along the contour 35 
for as far as the contour can continue to intersect the 
circle at exactly two points while maintaining an equal 
area on either side. If in so moving the contour, a location 
is found where further movement would cause the con- 
tour to intersect the circle at only one point, the contour 40 
nas ended, and the end point is designated as minutia 
point. This situation typically occurs only at the edge of 
a facial image and only rarely within the area ofthe face. 
If a location is found where further movement would 
cause the contour to intersect the circle at three or more 4S 
points, there is an inflection point within the circle. It can 
be located by considering the slope of the contour within 
the circle relative to the face axes. The point of maxi- 
mum change in slope is then designated as a minutia 
point. If a location is found where further movement so 
would cause the contour to intersect the circle at no 
points, there is a small island area within the circle. The 
centroid of that island is designated a minutia point. 

Referring now to Fig. 5. there is shown a facial im- 
age on which minutia points, e.g., 301 . have been iden- ss 
tified on facial thermogram 200, as described above. 



VII, Assignment of Characteristics to Minutiae 

Once minutia points are selected, minutiae gener- 
ator 104 assigns to each such point a label containing 
(X, y z, a, R. B, T), where x and y are the horizontal and 
vertical displacements of the point relative to the facial 
axes, z is the thermal value of the point a is the angle 
subtended by a tangent to the thermal contour at the 
minutia point, R is the radius as discussed above, 8 is 
the number or value of the thermal band in which the 
point is located, and T is the threshold imposed (if any) 
as discussed herein. In some environments, not all of 
these characteristics will be used and in such situations 
they need not be assigned. However, in some applica- 
tions, these characteristics may advantageously be 
used for matching. 

It should be recognized that numerous variations in 
the operation and structure of minutiae generator could 
be used. For instance, minutiae that are simply centro- 
ids of areas of constant thermal values could be used. 
Depending on the resolution of the thermal imaging de- 
vice 102, on the order of 300 thermal contours may typ- 
ically be generated for a face, leading to 300 minutiae. 
This number of minutiae may be sufficient for identifica- 
tion purposes, depending on the application and envi- 
ronment in which system 100 is used. In some applica- 
tions, it may be of interest to identify faces seen in 
crowds, or faces turned at any angle. Particularly in 
those applications, a significant number of minutiae 
points should be extractable so that even a partial face 
can be used for identification. 

As another possibility, only centroids located near 
the center of the face, or in concave areas of the face 
less vulnerable to artifacts due to edges, could be used. 
In other applications, minutiae may be derived using 
centroids of images where all thermal values less than 
a threshold are maintained, but those higher values are 
collapsed into one band. Centroids may be added to the 
set as the threshold is reduced. In such instance, each 
minutia point is characterized using at least the (x, y z, 
T) factors mentioned above, where T is the threshold. 
In a variation on this minutiae generation technique, in- 
flection points produced from such thresholding rather 
than the centroids may be used. 

Still another variation is to use start and stop loca- 
tions from run length encoding of facial thermogram 200 
to provide start and stop locations for thermal contours. 
Each stop/start location provides a minutia point which 
is characterized by the (x.yz) values discussed above. 

An additional approach is to designate undefined 
locations generated by compression and subsequent 
expansion of the image. Specifically, facial thermogram 
200 is compressed using wavelet or fractal-based meth- 
ods and then expanded again. Because such compres- 
sion techniques are lossy in a deterministic way, a mi- 
nutia set may be defined as the undefined locations re- 
sulting from a comparison of the original image with the 
compressed-and-expanded image. This approach pro- 



10 



19 



EP 0 867 830 A2 



20 



vides an additional advantage of compressing the data 
used lor recognition. 

The wide variety of techniques for generating minu- 
tiae described above provides an added measure of se- 
curity, as one attempting to mimic thermal facial minuti- s 
ae may be able to do so if one technique for generating 
minutiae is used by minutiae generator 104, but not if 
another is used. Thus, without prior knowledge of the 
particular technique being employed by minutiae gen- 
erator 104, system 100 becomes even more difficult to io 
comprise than it otherwise might have been. 

As mentioned above, it may be desirable that alt 
thermal images be scaled to a standard size prior to 
processing, it also may be desirable, depending on the 
thermal imaging system used, that all thermal images is 
first be normalized to a standard thermal profile before 
processing. In alternate embodiments, intended for var- 
ious applications and various environments, these pre- 
processing steps may significantly increase accuracy in 
recognition or may merely impose unnecessary 
processing overhead. For example, if system 100 is 
used in connection with an outdoor automated teller ma- 
chine, thermal normalization may be needed to deal with 
seasonally wide variations in surface skin temperature. 

25 

Minutiae Matcher 106 

As mentioned above, minutiae generator 104 and 
minutiae data generator 105 are used to produce minu- 
tiae data signals for a population of known persons. The 30 
data corresponding to these signals are stored in minu- 
tiae database 1 1 2. Thermal imaging device 1 02 then ob- 
tains a thermal image of an unknown person 101 and 
minutiae generator 1 04 produces signals representative 
of the minutiae and minutiae data generator 105 gener- 35 
ates data for the minutta for that person. Once these sig- 
nals have been produced, minutiae matcher 106 com- 
pares the signals representative of person 101 to sig- 
nals from minutiae database 102 corresponding to mi- 
nutiae data of known persons. In a preferred embodi- 40 
ment, minutiae matcher 106 performs three basic func- 
tions to obtain a match: alignment of the unknown face, 
comparison of minutiae data, and selection of a match. 
Each of these functions is described in greater detail be- 
low. 45 

I. Alignment of Unknown Face 

Because there may not be control over the position 
of the face of person 101 with respect to the field of view so 
of thermal imaging device 102 when image is obtained, 
the orientation of the face may not be such that the facial 
axes are aligned to be horizontal and vertical. Thus, mi- 
nutiae matcher 106 corrects the orientation by rotating 
the image such that the facial axes are horizontal and ss 
vertical. Next, conventional processing using a three di- 
mensional model is applied to correct for any rotation or 
twist of the head. In a preferred embodiment, such 



processing models the head as a sphere with a diameter 
equal to the apparent width of the face, and anti-distorts 
the image to provide a view which is normal to a surface 
plane across the forehead and upper lip and in which 
the center of the sphere coincides with the face center. 
In a conventional manner, the nose and chin are ignored 
so as not to disrupt positioning of this surface plane. 

II. Comparison of Minutiae 

Comparison of the minutiae data of the unknown 
person 101 with minutiae data from known persons be- 
gins by comparing locations of such minutiae. First, the 
locations of minutiae for a known face are considered, 
and denoted as M(K)i. Next an allowed positional error 
e is selected, as is determined to be appropriate for any 
given environment in which system 100 is used. The mi- 
nutiae of the known face are then overlaid on the minu- 
tiae of the unknown face, denoted M(U)j . Any M(U)j that 
are not with e of one of the M(K)i are ignored. Any M{K) 
i which are not within e of on the M(tJ)j are ignored. This 
leaves a residual set of minutiae pairs. Ifthis set is empty, 
there is not a match between the two images. Otherwise, 
the characteristics of the corresponding points are com- 
pared. 

Depending on the application, any comparison 
technique that considers the characteristics (x, y z, a, 
R, B, T) listed above may be used to generate a com- 
parison metric. In a preferred embodiment, only the po- 
sitional differences are considered. 

The simplest decision technique is to set a minimum 
number of pairs of corresponding minutiae for a poten- 
tial match. If an unknown face and a known face exhibit 
a least the minimum number of corresponding minutiae 
pairs, they are considered to be a potential match. 

In an alternative embodiment, the Ax and Ay values 
for each pair of corresponding minutiae are determined, 
and the distribution of Ay with respect to Ax is then de- 
termined for the overall set of minutiae pairs. The stand- 
ard deviation of that distribution is then compared 
against a threshold standard deviation to determine 
whether a potential match exists. 

In still another technique, a new error measure e' is 
introduced dependent not only on location but on ther- 
mal value (z). Minutiae pairs are only considered if they, 
are within a certain thermal value difference Az as well 
as have locations within the distance error e, thereby 
satisfying new error measure e*. 

Further levels of decision requirements can similar- 
ly be added to produce the desired level of confidence 
in the match for the application at hand. Each possible 
comparison of the unknown face with known faces is 
performed, and then the known images are rank-or- 
dered according to the goodness of fit (e.g. closeness 
in metric) with the unknown face. 
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III. Selection of a Match 

Through experience with use of the system on new 
images of known persons, a threshold value is estab- 
lished to provide a desired ratio of false positive and 
false negative identifications appropriate to the particu- 
!ar application. In a preferred embodiment, both self-cor- 
relations of multiple images of known persons and 
cross-correlations of different known persons in the da- 
tabase are used to help establish this threshold. 

If only one known person meets the threshold re- 
quirement, that person is selected as the match. If no 
known person meet the threshold requirement, a failure 
to match signal is produced. If multiple known persons 
match the unknown person to within the threshold dif- 
ference, the best matching person is selected. Alterna- 
tively, if multiple images of the same known person are 
referenced in minutiae database 1 1 2, the person having 
the highest ratio of matches within the top number of 
best matches may be used. For instance, if there are 
ten images of each known person in database 112, the 
top ten matching images determined by minutiae com- 
parison are considered. The person who is associated 
with the most of the top ten is selected to be the match- 
ing person. Additional levels of decision requirements 
may be added, either in a simple manner or iteratively. 
with a determination after each level as to whether a 
match decision can yet be made. 

In an alternative embodiment, minutiae matching is 
performed using techniques disclosed in U.S. patent ap- 
plication No. 07/984,514. filed December 2, 1992, and 
U.S. patent No. 08/314,729. filed September 29. 1994, 
which is a continuation of U.S. patent application No. 
07/984,514, both of which are hereby incorporated by 
reference in this application as if the entire contents of 
each had been fully reproduced herein. In this alterna- 
tive embodiment, flash correlation is used to match mi- 
nutiae through a digitized artifact-producing technique. 
In this embodiment, the size of a minutia point is prefer- 
ably increased to represent the possible error in its lo- 
cation, and minutiae are replicated successively along 
the face axes to increase their density and thereby in- 
crease the discernability of the correlation artifact that 
indicates a match between two images being compared. 
Such artifact is found to occur if any only if there is a 
match between two pixelized images. 

Other known matching techniques may alternative- 
ly be used in minutiae matcher 106, with tolerances es- 
tablished for errors due to imperfect knowledge of head 
position or distance, errors introduced by considering 
the head or face as a two dimensional surface or as a 
sphere, and other systematic and random residual er- 
rors. Some known fingerprint matching techniques may 
also be adapted to use with minutiae matcher 106. By 
analogizing thermal contour to fingerprint ridges, the 
multiplicity of facial thermal contours may be treated in 
a manner similar to matching many fingers per person. 
Alternatively, specific areas of the face, such as sur- 



rounding the canthi, may be selected and used alone 
for identification. Depending on the resolution of the 
thermal imaging device 102, several hundred minutiae 
may be extracted from a facial thermal image. As noted 
s above, lack of prior knowledge of which facial features, 
and which specific matching techniques are used for 
any particular application by system 100 increases the 
security of system 100 against being compromised by 
third parties. 

^0 For applications of system 100 to environments 
where legal proof of identification is important, a classi- 
fication scheme for faces may be useful, as fingerprints 
traditionally have been classified into various classes for 
such applications. For example, whorls, arches, and 
loops are conventional descriptors applied to ridges in 
the center of a finger. 

Another approach to classification of facial thermo- 
grams relates to obvious characteristics for use in ver- 
bally describing a given facial thermogram. Such char- 

20 acteristics include whether the canthi are merged or 
separated; whether the thermal contour of the nose is 
relatively cold, hot. or normal; whether the nose is trap- 
ezoidal in shape or irregular in shape; the degree of ther- 
mal symmetry of the forehead; and the degree of sym- 

25 metry in location of thermal features in the mouth cor- 
ners, the inner curves of the cheeks, the nose, the can- 
thi, and the outer corners of the eyes. To be useful, such 
designations should remain consistent over variations 
in imaging equipment, environmental conditions, phys- 
io iological variables, and other sources of errors. Accord- 
ingly, classification should not rely on features deter- 
mined to be highly sensitive to such factors. Classifica- 
tions based on overall image, e.g., those based on some 
of the distances between the features discussed in con- 

35 nection with Fig. 4 may be suitable for use. 

Another approach is to use wavelet coefficients that 
produce the minimum difference between an interpolat- 
ed wavelet-compressed imtage and the original image. 
Depending on how many classes are desired, that 

40 number of wavelet coefficient sets can be generated. 
Each image to be classified is compressed and then re- 
stored using each of the sets. The image is assigned to 
the class corresponding to the set of wavelet coefficients 
which best restores the image to its original form. 

45 Referring now to Fig. 6, classification of faces is 
achievable based on the number of minutiae, their char- 
acteristics, and their distribution over the face. A pre- 
ferred classification method segments the face repre- 
sented by thermogram 200 into a grid 401 of cells 402. 

50 Each cell is then classified based on the number of mi- 
nutiae located therein. As an example, the facia! ther- 
mogram 200 of Fig. 6 is divided into a grid 401 , the cells 
402 of which might be characterized as type A if the cell 
contains less than 3 minutiae, type B if the cell contains 

55 between 3 and 5 minutiae, and type C if the grid contains 
more than 5 minutiae. A face can then be classified 
based on the number of cells of each type that are found. 
For instance, one classification scheme is based on the 
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number of type A and type C cells. If a face is divided in 
to 36 grid celts as illustrated in Fig. 6, classes could be 
designated as nAmC. where n is the number of type A 
cells m is the number of type C cells, n+m = 36 - p and 
p is the number of B cells. Using this arrangemen t. 1 260 
classifications are possible. Alternatively, ranges of val- 
ues can be considered to be within the same class. 

As a further refinement to such classification, the 
degree of bilateral symmetry in distribution of type A 
cells and type C cells could be considered. If the face is 
divided into four quadrants designated upper right, low- 
er right, upper left, lower left, each quadrant having 9 
cells, a metric for classification could look at differences 
in the numbers of type A and type C cells in horizontally 
or vertically adjacent quadrants. Such metrics may be 
the absolute difference in minutiae between such quad- 
rant pairs, or may be simplified by merely indicating 
whether a left (or upper) quadrant has more, fewer, or 
equal minutiae as a corresponding right (or lower) quad- 
rant. 

Other possible classifications are based on geo- 
metric values of, and ratios among, the points and lines 
described in connection with Fig. 4, once the face has 
been normalized as described above. In some applica- 
tions, a combination of visual and thermal attributes may 
be employed for classification. For example, a ratio be- 
tween the distance between left and nghl canthi centro- 
ids 201 , 202 in facial thermogram and the distance be- 
tween the left and right pupils as determined through 
visual imaging is found to be a useful metric for classi- 
fication, as is the ratio between the distance from eye 
line 206 to horizontal central line 210 and the distance 
from a line connecting the eyes to the tip of the nose as 
determined by visual imaging, as is the ratio between 
the distance between left and right nostrils 203, 204 and 
the distance between the outer limits of the nostrils as 
determined by visual imaging. 

The usefulness of facial thermal imaging in recog- 
nition applications is increased by appropriately encod- 
ing thermal facial images so that consistent codes are 
generated each time a facial thermogram of a person is 
obtained. Such a coding scheme reduces database 
search and minutiae matching overhead, thereby allow- 
ing faster processing using less expensive equipment. 
In a preferred embodiment, overlaying a grid on a face 
such that 144 cells cover the area of the face, and as- 
signing a binary code to each cell, such t hat the cell is 
encoded with a T if the cell contains one or more mi- 
nutiae and "0" if the cell does not contain any minutiae, 
is found in practice to yield good results. Since this en- 
coding scheme preserves the relative location of each 
bit, it is straightforward to ignore selected bits in cases 
where only a portion of a face is imaged, due to obstruc- 
tion, disguise, or orientation. 

Use of such a "facecode" also facilitates straightfor- 
ward verification and comparison techniques. In some 
verification applications, for example, a requirement that 
10% of the coded bits match may be considered suffi- 



cient to provide a desired level of confidence. Simple 
difference comparison on a bit-by-bit basis, which is 
computationally extremely efficient, is sufficient to de- 
termine the number of corresponding bits between a 
_ 5_ code of an unknown face and that of a known face. 
Where multiple known faces exceed a threshold level of 
similarity, the one with the greater number of common 
bits is readily selected as a best match. 

Although the discussion above has been directed 
10 to thermal images of faces, it should be recognized that 
similar techniques and systems may readily be applied 
to Images of other body parts in accordance with the 
present invention. It should also be recognized that nu- 
merous other imaging modalities besides thermal imag- 
es ing may be employed in accordance with the present 
invention, for example x-ray, NfVIR, MRI, and CAT scan 
imaging. It should also be recognized that known 
schemes for pattern recognition and graph matching 
may be applied readily in accordance with the present 
20 invention, depending on the needs of a particular appli- 
cation. 

Standardized Infrared Minutiae Co-ordinate System 
fSlMCOS) 

25 

The method and apparatus described above for fa- 
cial minutiae extraction can be used to develop a stand- 
ardized minutiae co-ordinate system for identification of 
medical patients and for diagnosis of medical condi- 

30 lions. Because an infrared camera operates at a dis- 
tance from the patient and detects and records only ra- 
diant heat spontaneously emitted from the body surface, 
it constitutes a painless, non-invasive, passive method 
of recording patterns of body surface temperatures. 

35 These patterns have been found to depend upon the 
underlying vascular structure and are unique for each 
person. Infrared identification therefore provides a 
method for uniquely identifying individuals under all 
lighting conditions, including total darkness. It is not 

"to prone to forgery or multiple identity deception and so 
provides convenieni and highly secure identification of 
individuals. The method for generating repeatable reg- 
istration points on the skin surface of the human body 
utilizes discrete minutiae points obtained from the ther- 
mal images. Visual characteristics of the body, such as 
size and shape and relative position of body parts, are 
maintained in the infrared image. In addition, the details 
of the vascular system are indicated by the distribution 
of temperature across the skin surface. Current infrared 

50 cameras are sufficiently sensitive to temperature varia- 
tions that they clearly distinguish the skin directly over- 
laying blood vessels due to the thermal difference 
caused by the flow of warm blood. The vascular struc- 
ture appears as a white (hot) overlay of the circulatory 

55 structure on top of a grey scale image of the thermal 
map of the body, as shown in Fig. 7. 

In Fig. 8. there is shown apparatus 100a for 
processing infrared images to yield repeatable minutiae 
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points corresponding to specific vascular locations un- 
der the skin. The apparatus includes a thermal imaging 
device 102 for producing a thermal image I. A minutiae 
generator 104 and minutia data generator 105 are part 
of the program memory 1 08 as is a minutiae overlay de- _s 
vice 1 1 6. The set of minutiae obtained from any extend- 
ed area of the body is unique to each individual. In par- 
ticular, facial minutiae are unique between identical 
twins. The same thermal minutiae are repeatedly ex- 
tracted from a given individual. They are overlaid by the 
overlay device 116 and annotated by an image proces- 
sor 1 1 8 on the infrared image or on a visual or any image 
obtained from another medical sensor having the same 
orientation to the subject. From the processor, the an- 
notated image signal is delivered to an output display/ '5 
processing device which produces the thermal image 
with overlay lO. 

In Fig. 9, there is shown a modified apparatus 100b 
to that of Fig. 8, wherein a reference minutiae identifier 
120 is provided between the minutiae data generator 
1 05 and the minutiae overlay device 1 1 8 to identify and 
specify reference points 151 in the thermal image and 
overlay 10, The reference points allow manual or auto- 
mated comparison, merging, or registration among a set 
of images taken at different times with different orienta- 25 
tionsor different medical instruments. Figs. lOa-c illus- 
trate minutiae automatically extracted from a facial ther- 
mogram as the head turns. Current infrared cameras 
commonly produce 30 frames of video output per sec- 
ond, and minutiae extraction and annotation can be per- 3o 
formed in real time as those frames are generated. 

Fig 11 illustrates alternate apparatus 100c for gen- 
erating a medical image Ml of a portion of the body via 
a medical imaging device 1 22 such as an x-ray machine. 
The medical image can be annotated with a minutiae 3S 
overlay to generate a medical image with minutiae over- 
lay MIO from the output display/processing device. 

The inventive technique differs from visible recog- 
nition approaches in that it does not merely sample a 
finite number of points on an image; it extracts points 40 
which have particular meaning. This provides increased 
resolution at the same time it reduces the degree of 
computation required. The essential features of the 
technique are the uniqueness and invartance of thermo- 
grams, the use of a passive imaging technique to obtain 
subsurface details , the use of automated minutiae ex- 
traction to match different images taken of the same in- 
dividual, and the use of standardized minutiae locations 
to compare different persons or the same person as he 
grows from childhood to adulthood. The matching tech- so 
nique involves developing sets of corresponding points 
in two images, morphing one image into the reference, 
and measuring the degree of morphing as an indicator 
of the amount of difference. Similar automated proce- 
dures are then used to verity that imagery is from the 55 
same patient, identify a patient by comparison to a da- 
tabase of images, and compare images taken at differ- 
ent time and/or with different sensor modalities. 



The minutiae extraction and annotation procedure 
locates the position of each minutia. In addition, it may 
note characteristics of each point such as a vector indi- 
cating the orientation of the corresponding blood vessel, 
a second vector indicating the relative orientation of the 
branching blood vessel, normalized apparent tempera- 
ture measure, and apparent width of the corresponding 
blood vessels. As with some of the fingerprint minutiae 
matching machines use of the characteristic data can 
enhance the speed and accuracy of identification, Fur- 
themiore, It can improve the accuracy and speed of au- 
tomatic fusion of medical imagery. 

This basic technique can be employed on an area- 
by-area basis when portions of the body cannot be seen 
or when significant changes have occurred in portions 
of the thermogram such as when portions of the body 
have suffered external wounds. This would be done by 
segmenting the thermogram to consider only the por- 
tions of the body in which the minutiae can be detected. 
Functionally, this is equivalent to matching a latent par- 
tial fingerprint found at a crime scene to a full rolled print 
filed in the FBI system. The set of minutiae points, to- 
gether with characteristics which describe each such 
point and its relation to other minutiae is considered 
unique to the individual and persistent, for both contact 
fingerprints and thermal minutiae. 

Verification that two medical images are from the 
same person can be an end goal in itself or the first step 
in further processing the two images to extract compar- 
ison data. Telemedicine applications, electronic filing 
systems, insurance claims processing, updating of 
medical records, and extraction of medical histories dur- 
ing emergency treatment are some of the situations in 
which it is essential to have a reliable, fool-proof method 
for positive identification of the subject and precise lo- 
calization of imaged areas. 

The use of infrared identification (IRID) has several 
advantages over other methods for recognition of per- 
sons. IRID operates regardless of lighting conditions. It 
requires only a single frame of imagery, taken in 1/30 of 
a second, for positive identification, and so can provide 
on-the-fly recognition during emergency admissions or 
evacuations. The imagery can be collected at a dis- 
tance, without causing a delay or inconvenience to the 
subject. No parts of the system come into contact with 
the subject. Since thermal images are essentially im- 
mune from variations caused by illumination and shad- 
ows, it is not necessary to control those variables. IRID 
provides continuous identification and confirmation ver- 
ification of ID even in the dark. A cheaper, cruder form 
of thermal imaging can be obtained from use of heat- 
sensitive crystal sheets, which require contact with the 
skin. More expensiie, active imaging of the vascular sys- 
tem can be obtained from laser doppler. Either of these 
could be the source imager for extraction of minutiae; 
however, the operational advantages of passive thermal 
imagers make them the preferred sensor. 

Thermal minutiae can be obtained from commer- 
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cially-availabie thermal imaging devices sensitive in the 
3 to 5 or 8 to 12 micron wavelength bands. Images are 
this type are shown in Figs. 7, 10, 12, and 13. 

Current infrared cameras produce a standard ana- 
log or digital output providing 30 frames per minute as_ 
shown in Figs. 7-10. Tracking the minutiae from frame 
to frame assists in the exploitation of the dynamic IR im- 
agery by allowing measurements to be made over time 
from the same body locations while accommodating 
changes in position due to respiration, voluntary or in- 
voluntary movements of the subject, and intentional or 
accidental variation in the position of the imaging sys- 
tem. The use of infrared video imagery also allows the 
imagery to be recorded in real time for later analysis, 
and provides a self-documenting chain of custody iden- 
tification of the person recorded, ail without the neces- 
sity for the cooperation of the person being imaged. 

Infrared imaging can be used to locate minutiae 
points over the entire body surface which correspond to 
intersection points and branch points of the underlaying 
blood vessels. This provides a built-in set of registration 
points on the body's surface, which can be annotated 
onto images produced by any medical sensor used in 
conjunction with the thermal imager. The registration 
points then can be used to compare and combine med- 
ical images taken with different equipment at different 
times and under different conditions, facilitating compar- 
ison of those images. Also, the minutiae points provide 
reference points for continuous re-afignment of surgical 
instruments, radiation sources, and other diagnostic or 
treatment equipment. Since the infrared camera is to- 
tally passive, it can be used continuously during other 
medical procedures to overlay precise registration 
points on the other images while also monitoring for 
overheating, shock, hypothermia, renal failure, and oth- 
er medical conditions. At the same time, the pattern of 
minutiae points superimposed on each image provides 
positive identification of the patient. Such applications 
are of particular importance during telemedicine proce- 
dures. 

The normal body is basically thermally bilaterally 
symmetric. Side to side variations are typically less than 
0.25 degrees Celsius. This fact is used in assigning axes 
tothe body's image. Where the skin surface is unbroken, 
there is gradual variation of temperatures across blood 
vessels, with the highest temperatures across the body 
surface being directly on top of major blood vessels. Ma- 
jor thermal discontinuities occur at entrances to body 
cavities such as the eye sockets, nostrils, or moth. 
These provide global reference points for automatic ori- 
entation of the thermal image. Local and relatively minor 
discontinuities in the skin surface occur at scars, moles, 
burns, and areas of infection. The thermal surface can 
be distorted through pressures and activities such as 
eating, exercising, wearing tight hats and other clothing, 
sinus inflammation, infection, weight gain and loss, and 
body position. However, the minutiae points rennain con- 
stant with respect to their position relative to the under- 



lying blood vessels. 

The technique for thermal minutiae extraction and 
matching can be summarized as follows: 

_5 1 . Current thermal image is digitized. 

2. Current image is divided into pixels, where the 
size of the pixel relates to the resolution or quality 
of the result desired. 

10 

3. Certain pixels are selected as minutiae points. 

4. Each minutia is assigned a vector having magni- 
tude and directional information in relation to the 

IS surrounding characteristics of the thermal image. 
Additional characteristics, such as type of minutia 
may also be recorded for each. Typically for each 
whole body thermal image, there would be on the 
order of 1200 minutiae. 

20 

5. Set of minutiae vectors of the current image are 
compared by computer to the set of vectors of other 
images. 

25 6. Comparison results are used to determine corre- 
sponding minutiae from the two images and to 
morph or mathematically adjust one image with re- 
spect to the other to facilitate comparison. 

30 7. Differences between the current image and da- 
tabase images are computed for either the entire 
image or for areas of interest. 

It is desirable that all thermal images in a database 

35 be normalized to a standard thermal range and be 
seated to a standard size during search and comparison 
procedures. Both normalization and scaling eliminate 
some minute amount of identifying characteristics of a 
particular person or his condition. However, the stand- 

40 ardization procedures greatly aid in the exploitation of 
the database by reducing the need to calibrate every 
imaging sensor used to produce images which will be 
filed in, or compared to database images. For example, 
in accident triage with no accurate ground truth refer- 

45 ence in the scene and possibly use of inferior quality 
imagers, standardization to constant size and thermal 
range is appropriate in order to match against database. 
Furthermore, standardization facilitates use of simulat- 
ed imagery for telemedicine and telesurgery applica- 

50 tions. For example, when incorporated into the military's 
automated battlefield medical pod. real time normalized 
thermal minutiae can be used to properly position injec- 
tions and application of external pressure to stop bleed- 
ing. 

55 In addition, standardizing database images facili- 
tates comparison of imagery during growth from child- 
hood to adulthood, compilation of medical libraries of im- 
ages from large number of people, and automated com- 
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parisons of current imagery against the vast libraries for 
diagnostic purposes. Standardization to a common use 
of thermal minutiae would also provide a common ref- 
erence for comparing images obtained from different 
sensors which produce different resolution images. 5 

There is great utility to maintaining a summary med- 
ical record which could be carried in encrypted form on 
a small token or card. It would include important medical 
histor/ information, and would provide linkage into da- 
tabase holding more complete information. The use of io 
thermal minutiae can be of assistance, since it provide 
a standardized technique for segmenting the complete 
body, using the thermal minutiae as nodes on a grid of 
finite elements. The resulting cells in the grid would be 
coded based upon the compilation of all medical history ^5 
data relating to that area of the body, and compared 
against the standardized imagery and status of the cor- 
responding cell in reference model. If there were no en- 
try throughout the medical record for any imagery, diag- 
nostics, treatment, or injury involving that area of the 20 
body, or no significant deviation from the reference mod- 
el, there would be no data for the cell. If known standard 
condition involving that area were known, then the 
standard code for that condiiion would be entered. Other 
cooes would pertain to unknov/n conditions, continuing 25 
treatment, previous conditions successfully treated, etc. 
Conditions which are not localized such as high blood 
pressure, would by convention be assigned to specific 
cells within the body outline. 

The identification techniques set forth herein can be 30 
used to diagnose and monitor treatment for burn vctims, 
for stroke diagnosis, and in telesurgery and telemedi- 
cine. 

Blood vessels that carry nutrients to the skin are de- 
stroyed when tissue is burned. A high powered laser can 35 
be used to remove the burnad skin, leaving the healthy 
skin intact. Laser light is differentially absorbed and re- 
flected by live and dead skin, such as in the area of a 
burn. Certain dyes such as indocyanine fluoresce when 
in contact with laser light and can be injected into a pa- 
tient's blood to indicate healthy tissue. Alternately, an IR 
imager can be used to indicate dead skin vs. healthy 
skin, since the dead skin appears relatively cold as a 
result of having no functioning blood vessels. More than 
100,000 person per year in the US alone suffer severe ''^ 
burns. The total cost of treating these patients exceeds 
$2 billion. Treatment includes massive transfusions to 
replace the blood lost during surgery to remove dead 
skin. Blood loss is the main cause of death in burn pa- 
tients. Continued heavy bleeding often prevents a sue- so 
cessful skin graft after the burned skin is removed. 

Ideally perlonned, laser ablation kills a 100 microm- 
eter-thin layer of skin below the burned area. Due to the 
thinness, nutrients can still get through, however, the 
layer stops the bleeding from preventing a good graft. 55 
IR minutiae can be used to reposition the patient, mon- 
itor healing in specific areas, compare various salves, 
dressings, etc used, and re-photograph the person, 



achieving standard imaging results regardless of the 
skin tone of the patient. 

Referring now to Fig. 14. there is shown an appa- 
ratus for maintaining the position of a surgical instru- 
ment lSO relative to a surgical site in a patient 1 62 during 
a surgical procedure. A thermal imaging device 102 
such as an Infrared camera, generates a thermal image 
output to a minutiae generator 104 to identify minutiae 
in the vicinity of the surgical site. As described above 
with reference to Fig. 9. a reference minutia pattern is 
also generated for a statutory patient. A surgical instru- 
ment position detector 166 senses the position of the 
instrument and produces an output signal correspond- 
ing therewith. The minutiae pattern for the patient and 
the surgical instrument positions signals are delivered 
to a comparator 168. The relative position of the instru- 
ment to the surgical site is set as a reference at time 0 
at 170. Displacement of the patient and/or the instru- 
ment with respect to the reference at times other than 0 
is detected by the comparator which can produce an 
output 172 used to reposition the instrument via a device 
1 74 for proper orientation to the surgical site. 

Each year. 500,00 American have strokes and 
150.000 of them die, making stroke the third leading 
cause of death , and the major cause of disability among 
adults. In the near future, new drug therapies may be 
able to return blood flow to stroke-damaged (ischemic) 
tissue, protect it from permanent damage, and promote 
recovery of function. However, the primary care physi- 
cian must be able to characterize the patient's acute 
neurological injury precisely enough to guide laboratory 
assessments and treatments. Stroke is a heterogene- 
ous group of conditions with many causes, levels of se- 
verity, and clinical presentations. 

Identifying the point of stroke damage and the ex- 
tent of damage depends on a pattern of normal and ab- 
normal findings. Computer tomographic (CT) findings 
often are normal during the first hours after ischemic 
stroke. Also, abnormalities found with neuroimaging 
may be unrelated to the patient's acute problem. Further 
confusing the situation, seizures, tumor, and intercraniat 
hemorrhage can mimic stroke. Treatment with anticoag- 
ulants or experimental clot-dissolving agents would be 
contraindicated in patients with hemorrhage but could 
be of value in some patients with ischemic stroke. 

Continuous IR monitoring of the patient can possi- 
bly assist in detecting and tracking minute variations in 
blood flow patterns associated with the onset, location, 
and severity of stroke; and also associated with reaction 
to drugs and other treatments. 

Within the next 10 years, both military and civilian 
medicine are expected to make routine use of telesur- 
gery where the patient and surgeon are not co-located. 
It is estimated that 90% of the information a physician 
needs to know about a patient can be acquired and 
brought to him electronically. In laparoscopic surgery to- 
day, a surgeon looks only at video images without look- 
ing at the real organs at all. Dermatology and pathology 
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are both already using electronic images as well as x- 
rays and medical records. Laparoscopic surgery is an 
electronic form of surgery. Teleradiology. telepathology, 
and teleconsulation are already, widely accepted elec- 
tronic medical practices. 

As telemedicine and telesurgery become more 
common, there will be more potential for error in identi- 
fication of patients and the treatment to be performed, 
and more need to document the precise medical history 
and treatment procedures performed by a given doctor 
on a given day. Filing, recall, and comparison of docu- 
mentation collected over time by different sensors at dif- 
ferent facilities will need to be automated to a greater 
degree, while protecting the privacy of the patients. The 
identification technique of the invention offers a low- 
cost, repeatable, non-invasive, passive system for 
standardization and registration of many current forms 
of medical imagery, while also offering an approach to 
high security maintenance of files with immediate ac- 
cess in emergency situations. 

Identification of Drug and Alcohol Usage 

Many drugs, including cocaine and alcohoL are va- 
soconstrictive substances which cause cooling of the 
skin surface, the resultant cooling is detected through 
passive imaging of the thermal energy emitted form the 
face. In Fig 1 5 is shown the thermal image of an individ- 
ual who is substance free and in Fig. 16 is a thermal 
image of the same individual after the ingestion of alco- 
hol. 

The thermal imaging techniques of the present in- 
vention can be used to detect substance use by individ- 
uals, even where the individuals's identity is unknown. 
This is accomplished by compiling databases of statis- 
tical analysis of thermal signatures obtained from clini- 
cal trials in which cooperating subjects have concurrent 
drug testing performed using urinalysis or blood testing 
along with thermal signatures obtained from known sub- 
population but without concurrent testing by other 
means. 

The vascular system supplying the human face typ- 
ically exhibits thermal variations on the order of TO 
across the facial surface. Certain general features, such 
as hot patches in the sinus areas, relatively cool cheeks, 
and cold hair pertain to all facial thermograms. Other 
features such as specific thermal shapes in certain ar- 
eas of the face are characteristic of a particular person. 
Measured disturbances to other features, such as the 
general symmetry between two sides of face, range of 
thermal variations in the forehead, peak temperature, 
size of canthi pattern, and variations in those disturbanc- 
es over time, may be correlated with a high probability 
of drug or alcohol use. 

Variations in temperature across the facial surface 
can be imaged by thermal cameras sensitive to wave- 
lengths in the 3-5. 8-12, or 2-15 micron ranges. Current 
cameras can provide thermal resolution better than 



CO/^C and spatial resolution of better than 02", result- 
ing in 65.000 to 265.000 discrete thermal measure- 
ments across the surface of the face. For most such 
cameras, that thermal map is regenerated 30 times per 
5 second to produce either a standard video output which 
can then be recorded and processed on standard vide- 
otape equipment, or a direct digital signal which can be 
immediately input to a computer. 

Certain drugs appear to produce characteristics 
TO features in facial thermograms, which may be identifia- 
ble from detailed analysis of the structural patterns and 
distribution statistics. Furthermore, the rate of change 
at any point in time may be a discriminator between 
chronic and recent use of each drug. Using currently 
'5 available thermal imaging cameras, thermal signatures 
emitted from the face can be used to deduce changes 
in activity levels of specific arteries in the brain which 
are known to be affected by particular drugs. 

The vascular system has a common structure in 
20 each person, with known pathways for instance from the 
heart to the brain, and known pathways between blood 
vessels in the face and those in the brain as shown in 
Figs. 2b and 2c. Using the SIMCOS technique, a set of 
standardized minutiae appearing in the face can be 
^5 identified. Through clinical drug trials using known 
types, amounts, purity, and administration techniques, 
the thermal effects over time at each such minutiae lo- 
cation can be observed. The effect of varying the type, 
amount, or purity of drug can also be observed. The ef- 
30 feet on different people can be obsen/ed. Since the ther- 
mal effects may be quite small and localized, it is impor- 
tant to utilize the SIMCOS method for identifying the pre- 
cise minutiae locations in each subject. That provides 
repeatability of measurements over time without requir- 
es ing the application of registration markers to the face, or 
the use of invasive techniques to repeatably find the 
same locations. Also, it provides a method for compar- 
ing corresponding locations in different subjects. 

Statistical analysis of time-varying thermal signa- 
ge tures at each facial minutiae point before, during and 
after drug or alcohol administration provides a reference 
dataset which represents the thermal effect of that sub- 
stance uncjer the protocol used. A library of thermal mi- 
nutiae substance effect signatures can be developed for 
45 various drugs and other substances for which screening 
is desired, including for prescription and over the coun- 
ter medications, tobacco, and alcohol. Fig. 17 presents 
an illustration of thermal signatures associated with sub- 
stance-free subjects. These may differentiate between 
so sex, age, size, medical history, or other characteristics 
of the substance-free subjects. In addition, thermal mi- 
nutiae on non-substance signatures may be developed 
for each person enrolled in to a system who will be sub- 
sequently scanned for substance use. At the time that 
55 the thermal data is collected, urinalysis or blood testing 
can be performed to assure that the subject is sub- 
stance-free. 

Subsequently, each time a person enrolled in the 
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system is scanned, the same facial thermal minutiae are 
located and the corresponding thermal signatures com- 
puted. They are compared against that subject's own 
substance-free thermal signatures. If there is sufficient 
match, no further analysis is required. Otherwise, the s 
thermal signatures are compared against the library of 
substance effect signatures. If there is sufficient match, 
the system provides an output to the system operator 
that a potential substance detection has occurred. 

If desired, relevant data from the system may be io 
transmitted to a medical review officer for a final deter- 
mination. Such data could include the current thermal 
imagery of the person's face, reference imagery from 
the database taken of the same person when he was 
known to be substance-free, the thermal signature cal- '5 
culations which lead to the systems's determination of 
a possible substance detection, the library reference 
thermal signature which was used by the system to 
make that determination, the system's calculated confi- 
dence in the determination, thresholds which were set 20 
by the system manager relative to the infrared camera 
sensitivity and other system factors, medical data on file 
about the person including known or self-reported use 
of prescription or over the counter medication, or a past 
history of substance use. The Medical Review Officer 25 
determines whether to accept or override the system's 
determination, or specify that further testing such as uri- 
nalysis is to be performed. 

The technology of this patent involves clinical stud- 
ies in which known amounts of controlled substances 30 
are administered to subjects whose thermal images are 
monitored and stored to provide archival references 
Analysis of those images is used to determine the time- 
varying thermal effect of specific drugs at specific minu- 
tiae sites on the face, which are specified by anatomical 35 
landmarks. After an individual ingests a drug, changes 
in his or her thermal signature gradually occur until a 
thermal "climax" is reached after which the signature 
gradually returns to its normal state. In chronic drug us- 
ers, permanent physiological changes may occur such 
that there is no longer a smooth total decay of the ap- 
parent drug-induced effects. By processing a significant 
number of thermal images, thermal signature markers 
are identified and related to standardized vascular sys- 
tem locations whose thermal variation are highly corre- -^5 
lated with use of the particular substances. References 
may be developed for an individual for a class of indi- 
viduals grouped by age or other characteristic, or for a 
general population. 

A general determination of substance-free vs. sub- 5o 
stance-influenced classification may be based upon da- 
ta collected on the thermal effects of various substances 
of interest. In addition, certain substances produce char- 
acteristic results which may be identifiable from detailed 
analysis of the thermal signatures associated with facial 55 
minutiae, and/or with distribution statistics from those 
signatures. Furthermore, the rate of change at any point 
in time may be a discriminator between chronic and re- 



cent use of each drug. Techniques for processing se- 
quences of thermal images may enhance the visibility 
of bilateral asymmetries, anomalous static conditions, 
and unusual time-varying trends in the thermal signa- 
tu res ass ociated with specific minutiae locations in the 
face to indicate activity levels of specific arteries in the 
brain which are known to be affected by particular drugs. 
Therefore, when a substance is known to affect partic- 
ular functions, vascular pathways to the corresponding 
brain areas should be analyzed for related thermal sig- 
natures at minutiae points along the pathways, under 
the assumption that increased activity at the brain site 
will be found to correlate with increased vasomotor ac- 
tivity along pathways leading to that site, as evidenced 
by thermal changes. 

In order to best compare images from different peo- 
ple and under different conditions, facial thermograms 
must be standardized and registered to common coor- 
dinates. The preferred approach is to use the standard- 
ized infrared minutiae coordinate system (SIMCOS) 
technique which locates standard minutiae points on 
each facial thermogram. In its preferred embodiment, 
the SIMCOS minutiae correspond to anastomoses 
which are connections or branchings of major superficial 
veins and arteries in the facial area. Approximately 1 75 
such points exist in the face. A subset of the SIMOS mi- 
nutiae which relate to blood vessel or areas of the face 
affected by a particular substances is selected. The sub- 
stance activist at the brain site wilt be found to correlate 
with increased vasomotor activity along pathways lead- 
ing to the site, as evidenced by thermal changes. 

The collection of differences between the time var- 
ying thermal signatures for the substance-active minu- 
tiae, compared to the collection of time-varyingng ther- 
mal signatures for the same points in the absence of the 
substance, represents the marker for that substance. 
For each substance of interest, a marker may be devel- 
oped for a particular individual, for a class of persons 
grouped according to some criterion, or for a more gen- 
era! population. The substance-free marker can likewise 
be developed for a particular individual, class of per- 
sons, or general population. 

In subsequent screening of a known individual for 
a particular substance, his current thermal image(s) are 
analyzed to extract substance-active minutiae which 
can be seen in the available image(s). The set of thermal 
signatures is compared to the substance marker collec- 
tion and substance-free marker collections. Measures 
of similarity are calculated for individual, class, or gen- 
eral population comparisons for each substance of in- 
terest. Various correlation associated with substance 
use, or with substance-free references in the system li- 
brary. Normalized temperatures vs. time waveforms for 
each minutiae can be compared with the corresponding 
waveform from the reference. The waveforms are slid 
along the time axis until the best fit is found, since it is 
not known when, if at all, the person being screened may 
have used that substance. Another correlation ap- 
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proach involves sampling the thermal waveforms and 
producing a matrix ol values, where one dimension of 
the matrix is the number of minutiae used, and the other 
is the number of temperature samples over time. The 
reference library can include wider matrices, involving 
longer time period than is practical for an operational 
screening system. The comparison between the collect- 
ed matrix and the reference matrix would use a digital 
shifting and difference calculation to find the best area 
of match. 

A measure of goodness of match is made between 
the collected thermal signatures and the signatures for 
each substance under each protocol in the library. The 
system manager selects a threshold to be applied to 
each comparison, such that matches which are closer 
than that threshold will cause the system to issue a no- 
tice of possible substance detection. 

The results of comparison with the different markers 
may be recorded or stored or output to decision mark- 
ers. Alternatively, thresholds may be automatically ap- 
plied to the calculated differences to render a pass/fail 
or clean/under-influence determination. The statistical 
estimate of confidence in the determination can also be 
presented 

The apparatus for drug and alcohol detection is 
shown in Fig. 1 8. Three primary functions are performed 
with the apparatus; Enrollment. Reference Signatures 
Development, and Screening. System components may 
in general be rendered as software, hardware, or 
firmware elements. 

Prior to automated operation of the Identification 
and Detection (ID&D) system, a human operator termed 
a System Manager must perform set-up and initializa- 
tion of the system, which he does via the System Man- 
ager Interface, which includes a monitor, keyboard, and 
possibly a printer and other peripherals such as a mouse 
which are normally associated with personal computers. 
A System Manager must confirm the identity of the en- 
rolfee 4, and input the associated identification informa- 
tion into the person identifier and condition identifier da- 
tabase 6 within the system processor 8. The enrollee's 
current and past medical history data is also input to the 
system, including the results of urinalysis or blood tests 
todetect substance use. current use of illicit substances, 
and other information which may bear upon alcohol and 
drug testing results. 

The enrollee stands or sits at designated location 
facing the infrared camera 12 and within reach of the 
event trigger 1 4. When the event trigger is engaged by 
either the enrollee or System Manager, the output from 
the infrared camera is sampled by the frame grabber 16 
and the resulting frame stored in its buffer. At the same 
time, the camera output is recorded on a video cassette 
recorder 18 which incorporates annotation of the date 
time, location, and identify of the enrollee. 

The image in the frame grabber and buffer is proc- 
essed by the face locator 20 which determines that the 
image includes a single face which is in focus and of a 



suitable size and position. If the image is not suitable 
according to software criteria established within the face 
locator, a new image frame is grabbed and the process 
repeated until a suitable image is obtained. 
^ The ima ge is then processed by the minutiae ex- 
tractor and identifier 22 which locates the SIMCOS mi- 
nutiae points and extracts their positions on the image 
and the corresponding apparent temperatures. Addi- 
tional frames are grabbed and processed for a period of 

10 time selected by the system manager. 

The extracted minutiae locations and correspond- 
ing temperatures are processed by the thermal signa- 
ture extractor 24 which generates for each minutiae 
point the thermal variation over time. For enrollment pur- 
rs poses, a single frame may suffice. However, multiple 
frames over a period of seconds should be taken in or- 
der to help calibrate and factor-out noise in the system. 

Personal data about the enrollee is transferred to 
the enrollment database 26, along with the thermal sig- 

20 natures extracted for the enrollee. If the enrollee is 
known to be substance-free, the thermal signatures are 
also transferred to the substance-free signature data- 
base 28 stored by enrollee and also to the substance- 
free thermal signature database stored by classification 

25 of enrollee 30. 

If the enrollee is known or found to be substance- 
influenced, the thermal signatures are instead trans- 
ferred to the substance-related thermal signature data- 
base stored by substance and also to the substance- 

30 related thermal signature database stored by protocol 
34. The definitions of protocols will relate to clinical trial 
used for developing reference thermal signatures, and 
may also include self-reporting classifications such as 
"heavy" regular user of cocaine", or "infrequent user of 

35 marijuana but not within the past month 

The extracted thermal signatures of the enrollee are 
also transferred to the thermal signature comparator 
and statistical analyser 36 which compares the signa- 
tures of the enrollee with other signatures in the data- 
bases. If the enrollee's signatures vary too much from 
the others in the same substance-free class or from oth- 
ers in the same substance-related or protocol-related 
databases, then the system manager may request re- 
view by the medical review officer 38 prior to including 

45 the enrollee's data in the database. 

If no anomaly is detected in the enrollee's thermal 
signature, then the enrollee is instructed as to how to 
activate the system for future access and screening. If 
a personal identification number is to be used, that PIN 

50 will be assigned. If voice recognition. ID card, or other 
technique for identification is to be used, those proce- 
dures will be taught. The enrollee is now enrolled in the 
system. 

The substance use identification and detection sys- 
55 tern requires databases of thermal signatures from sub- 
stance-free and substance-related images. During clin- 
ical trials in which substances are administered under 
rigorous protocols, the same apparatus may be used to 
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generate the reference databases. 

A clinical trials investigator will serve as system 
manager . He will perform set-up and initialization of the 
system, the system manager interface 2, which includes 
, a monitor or other display, keyboard, and possibly a 5 
printer and other peripherals such as a mouse which are 
normally associated with personal computers. The sys- 
tem manager must confirm the identity of the enrollee 
2, and input the associated identification information into 
the person identifier and condition identifier database 6 
within the system processor 8. Since the same person 
may be enrolled several times under various protocols, 
he is termed the "subject" to emphasize that he may 
have several separate files within the enrollment data- 
base 26. The subject's current and past medical history '5 
data is input to the system the first time he is imaged. 
During each different test involving different substances 
and/or protocols, the specifics of the protocol used, as 
well as recent medical data, including the results of uri- 
nalysis or blood tests to detect substance use, will be 20 
included in the person identifier and condition ID Buffer 
6. The contents of that buffer are transmitted and stored 
with the results of the thermal signature extractor 24. 

The extracted thermal signatures of the subject are 
transferred to the thermal signature comparator and sta- 25 
tistica! analyzer 36 which compares the signatures of 
the subject with other signatures in the databases, if any 
If the subject signatures vary too much from the others 
in the same substance-related or protocol-related data- 
bases, then the system manager may request review by 30 
the medical review officer 38 prior to including the sub- 
ject's data in the databases. 

The thermal signature comparator and statistical 
analyzer 36 processes the thermal signature to estab- 
lish composite signatures or common characteristics 35 
which represent each ofthe databases: substance-tree 
signatures for each person enrolled in the system; and 
substance-free signatures for each designated class of 
enrollees, such as: people under age 16, people 1 7-20, 
people 21-30, people over 30, cigarette smokers, non- 40 
smokers, people taking heart medication, diabetics on 
insulin, athletes, vegetarians, social drinkers, etc. If a 
sufficient number of subjects is used, likewise compos- 
ite signatures or common characteristics will be calcu- 
lated for each substance in the database of substance- 45 
related signatures, and for each protocol. 

After the end of clinical trials, frames are extracted 
from the archival videotapes and used in place of imag- 
es directly taken by the infrared camera. Each extracted 
frame is processed by the system, and the thermal sig- so 
nature comparator and statistical analyzer 36 selects 
the best matching composite signature and classifies 
the frame accordingly First, the thermal signature col- 
lection of that frame is classified as better matching the 
composite for all substance -free trials or the composite 55 
for all substance-related trials. Second, if the subject in 
the frame is judged to be substance-free, the best 
matching class of database 3 is then selected. If the sub- 



ject is judged to be substance-related, then the best 
matching substance of database 32 is selected. Third, 
in the substance-free case, the system identifies the 
best-matching enrolled person's signatures within the 
class-selected.-ln-the substance-related case, the sys- 
tem identifies the best- matching protocol for the sub- 
stance selected. Since the true classification of each 
subject on the video tape is known, the performance of 
the system in an automated mode can thus be deter- 
mined, and the error rates associated with the classifi- 
cation can be calculated. Improvements to the system 
can then be made according to standard techniques for 
statistical analysis, including the use of neural nets to 
adjust weightings consideration of the data from certain 
minutiae more or less than others, normalizing the ther- 
mal variations or not, and normalizing time intervals 
based upon the size or metabolism of each subject. 

When those system parameters have been adjust- 
ed to maximize the correct classification from the video 
tape archives, the resulting reference signatures are 
stored in database 38 and used to speed the screening 
of future enrollees during operational use of the system. 

An enrolled person seeking entry through the sys- 
tem as an entrant stands or sits at designated location, 
facing the infrared camera 12 and within reach of the 
event trigger 1 4. When he is in position, he activates the 
event trigger. The output from the infrared camera is au- 
tomatically sampled by the frame grabber 16 and the 
resulting frame stored in its buffer. At the same time, the 
camera output is recorded on a video cassette recorder 
18 which incorporates annotation of the date, time, lo- 
cation, and identity of the enrollee. 

The image in the frame grabber and buffer is proc- 
essed by the face location 20 which determines that the 
image includes a single face which is in focus and of a 
suitable size and position. If the image is not suitable 
according to software criteria established within the face 
locator, a new image frame is grabbed and the process 
repeated until a suitable image is obtained. 

The image is then processed by the minutiae ex- 
tractor and identifier 22 which locates the SIMCOS mi- 
nutiae points and extracts their positions on the image 
and the corresponding apparent temperatures. Addi- 
tional frames are grabbed and processed for a period of 
time set into the system by the system manager. 

The extracted minutiae locations and correspond- 
ing temperatures are processed by the thermal signa- 
ture extractor 24 which generates for each minutiae 
point the thermal variation over time. 

The entrants's ID is transferred to the thermal sig- 
nature comparator and statistical analyzer 36, along 
with the thermal signatures extracted for that entrant. 
The thermal signature comparator and statistical ana- 
lyzer compares the signatures of the entrant with data- 
base of reference signatures 38. The closest reference 
signature is selected and the entrant is designated to 
have the same classification. That classification is trans- 
mitted to the system manager interface, which may 
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grant access through manual intervention or through au- 
tomatic control of an access portal. Depending upon the 
particular classification selected, or the amount of vari- 
ation between the entrant and the selected reference 
signatures, the result classification may be sent either 
automatically or through manual intervention to a med- 
ical review officer 40 for a final determination. The en- 
trant's file from the enrollment database 26 is also sent 
to the MRO to provide background information. 

Statistical analysis of drug or alcohol use within a 
random population is also encompassed by the present 
invention. This is performed by scanning a crowd and 
locating faces therein for analysis. 

Various standard methods for locating faces in an 
image frame can be used. A particularly useful approach 
uses an ellipse detector to find relatively warm ellipses 
(thermal faces) within a relatively cool background. The 
ellipse is located within the expected height range for 
humans, the detected temperature is within the expect- 
ed range for human faces, and characteristics common 
to all facial thermograms (hot canthi regions, symmetry 
of the eyes, nostrils, ears, cheeks, etc.) are not violated. 
Each detected face in each frame is evaluated to deter- 
mine if it meets the quality requirements for further 
processing. Requirements include the facial image be- 
ing in focus, being large enough to provide adequate 
resolution of the facial minutiae, being oriented close 
enough to full face forward and being free enough of 
blockages including beards, eyeglasses, and interven- 
ing obstructions, such that a sufficient number of facial 
minutiae can be extracted from the facial image. The 
specific requirements are dependent upon what sub- 
stances are to be detected, in how small a dose, and 
after what period of time. Faces which do not meet the 
quality requirement are not further considered. Those 
which are qualified are assigned unique tags. 

The next frame is then processed and quality faces 
are detected as above. Each such face is then com- 
pared to the faces in the previous frame, or to those fac- 
es from the previous frame which are close enough in 
location that they could be a particular face in the current 
■frame. Matching is performed using the facial minutiae 
matching method. If a current and prior face are deter- 
mined to be the same, then they are given the same tag. 
This process continues with subsequent frames being 
likewise analyzed. In general, a particular tagged face 
will move across and then out of the field of view. When 
the face is no longer seen by the camera then the ther- 
mal signatures associated with each of its minutiae in 
each of the frames are combined and matched against 
a reference database for non-substance and sub- 
stance-related signatures. 

The system is designed for stand-alone operation. 
IT is deployable for programmable periods of time, dur- 
ing which it will analyze and classify each face which 
appears within its field of view. The system will not rou- 
tinely record or store the thermal images, although pro- 
visions are made todothat during testing and evaluation 



of the system in order to allow for improvements to be 
made in the system and compared with earlier results. 
The output from the system will be graphical results 
such as shown in Fig. 19. The cumulative detection in- 
^ ciex of the y- axis re presents the number of persons who 
the system estimates have used marijuana, cocaine, or 
heroin in an amount and within a timeframe which re- 
suits in a residual level indicated by the x-axis value at 
the time of the analysis. The x-axis represents the con- 

10 fidence level of drug signature indication, which is relat- 
ed to the detection precision of the testing and analysis 
procedures. Separate curves indicate the specific drugs 
detected, and a composite survey indicates detection of 
any of the substances. Due to the frequent use of com- 

'5 binations of drugs, the composite curve is expected to 
be more significant than its components. 

The system is tested using known populations of 
drug users, and its results compared to urinalysis re- 
sults. The comparison is used to select thresholds for 

20 system decisions on classification of thermal signa- 
tures. The system can be deployed within a high inten- 
sity drug trafficking area, and its results compared to 
other current estimates of drug usage in that area. 
The apparatus for statistical analysis is shown in 

2S Fig. 20. Three primary functions are performed with the 
apparatus: face acquisition and tagging; face analysis 
and classification; and statistical population analysis. 

An infrared camera 42 is positioned such that per- 
sons in the population to be scanned generally enter, 

30 transverse, and exit the camera's field of view 44. The 
infrared camera produces a sequence of frames using 
either direct digital output or frame-grabbed video output 
46, which is stored in the frame buffer storage 48. 
Processing of the imagery is initiated by an initializer 

35 mechanism 50 which may be a proximity detector, mo- 
tion detector, or other sensor used to detect the possible 
presence of humans within the field of view. 

The system processor 52 is comprised of five com- 
ponents. The face locator 54 applies rules to the buffer 

40 stored image to identify all faces in the frame. The face 
quality check 22 applies additional rules to determine if 
a given face provides sufficient information in terms of 
focus, resolution, position, and number of minutiae 
available. Each qualified face is tagged and then com- 

4S pared with qualified faces in the preceding frame, using 
the SIMCOS technique, to determine which faces have 
already been seen and tagged. 

Faces which are seen in a given frame may have 
been temporarily blocked in the previous frames, and 

so so any face may be blocked, turned, or otherwise un- 
qualified part of the time. Comparison against earlier 
qualified faces is continued for a period of time which is 
considered a reasonable maximum time for transvers- 
ing the filed of view. The face tracker 54 reassigns tags 

55 so that the same face receives the same tag in subse- 
quent frames. There are two reasons to be concerned 
about consistently applying the same tag to each face. 
First, drug and alcohol detection accuracy improves 
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when a face is analyzed for a longer period of time. Sec- 
ond, statistical accuracy improves when each person is 
counted only one time, regardless of their position or 
speed, or their being within a group. 

Once a face has exited the field of view, the thermal 
signature extraction processor 60 produces time vary- 
ing apparent temperature signatures tor each facial mi- 
nutia seen over the course of time of all frames in which 
that face was seen. The composite thermal signature 
for that face is produced at 62. 

The analyzer 64 of the composite thermal signature 
compares each imaged person's composite thermal sig- 
nature to a database of signatures associated with 
known substances and protocols, or with known sub- 
populations such as cocaine addicts or alcoholics. More 
particularly, a signature from a thermal signature data- 
base from deployment 66 is compared with a signature 
from the reference thermal signature 68 in a comparator 
70, the output of which is delivered to a thermal signa- 
ture classifier 72. The analyzer output is delivered to a 
statistical analyzer 74 for processing the thermal signa- 
tures by batch to produce an output summary to the out- 
put report generator 76 for the output device 78. A sta- 
tistical sulficiency analyzer BO evaluates the report and 
determines whether the analysis of the population up to 
that point is statistically sufficient for the intended pur- 
pose. If not, additional frames are analyzed, if the anal- 
ysis is sufficient, then the system is re-initialized by the 
initializer 50 starting a new collection of faces and lead- 
ing to a new statistical evaluation. 

If the population throughput is well-known, the sta- 
tistical analyzer 74 can be programmed to run for set 
period of time before the batch is considered complete 
and control is passed to the output report generator 76. 
Other criteria for determining a batch size can be used, 
including counting the number of tags assigned or the 
number of detected suspected drug users or the number 
of detected substance-free persons. 

While in accordance with the provisions of the pat- 
ent statute the preferred forms and embodiments ofthe 
invention have been illustrated and described, it will be 
apparent to those of ordinary skill in the art that various 
changes and modifications may be made without devi- 
ating from the inventive concepts set forth above. 

Claims 

1. A system for recognizing individuals, comprising 

(a) an imaging device for producing a first sig- 
nal representative of sensed characteristics of 
the individual; 

(b) a minutiae generator connected with said 
imaging device for receiving said first signal 
and producing in response thereto a second 
signal representative of minutiae of the individ- 



ual, the minutiae corresponding to specific 
branch points of blood vessels of the individual; 

(c) a minutiae database for storing minutiae da- 
^ ta characteristics of minutiae for each of a plu- 

rality of known individuals and for producing a 
third signal representative of minutiae of each 
of the plurality of known individuals; and 

^0 (d) a minutiae matcher connected with said mi- 

nutiae generator and said minutiae database 
for receiving said second and third signals and 
for producing a fourth signal when a match oc- 
curs between the minutiae for the individual and 

'5 for one of the plurality of known individuals. 

2. A system as defined in claim 1 , wherein the sensed 
characteristics are non-visual characteristics. 

20 3. A system as defined in claim 2, wherein the sensed 
characteristics are thermal characteristics. 

4. A system as defined in claim 3. wherein said imag- 
ing device comprises an infrared camera. 

25 

5. A system as defined in claim 3, wherein said minu- 
tiae are limited to those on the face of the individual. 

6. A method for recognizing an individual, comprising 
30 the steps of 

(a) sensing non-visual characteristics of known 
individuals; 

^5 (b) identifying minutiae of each of the known in- 

dividuals in response to said sensed non-visual 
characteristics thereof, said minutiae corre- 
sponding to specific branch points of blood ves- 
sels of the known individuals; 

40 

(c) sensing non-visual characteristics of a new 
individual; 

(d) identifying minutiae of the new individual in 
'^^ response to said sensed non-visual character- 
istics thereof; 

(e) determining a distance metric for each of the 
known individuals with respect to the new indi- 
go viduals in accordance with one of the identified 

minutiae and the data characteristics of the 
identified minutiae thereof; and 

(f) determining a match between the individual 
55 and one of the known individuals in accordance 

with the distance metrics. 

7. A method as defined in claim 6, wherein said non- 
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visual characteristics are thermal characteristics. 

8. A system as defined in claim 7, wherein said minu- 
tiae are limited to those in the face of the individual. 

9. A method for classifying an individual, comprising 
the steps of 

(a) sensing characteristics of the individual: 

(b) producing a normalized representation of 
the individual in response to the sensed char- 
acteristics thereof; 

(c) identifying minutiae of the individual in re- 
sponse to the normalized representation, said 
minutiae corresponding to specific branch 
points of blood vessels of the individual; 

(d) determining a correspondence between a 
grid of cells and the normalized representation; 
and 

(e) classifying the individual in response to co- 
location of at least one of said minutiae and at 
least one of said cells, thereby identifying the 
individual as corresponding to a class. 

10. A method as defined in claim 9, wherein said clas- 
sifying step includes encoding the identity of the in- 
dividual using a plurality of bits each corresponding 
to one of said cells, setting one of the plurality of 
bits to a first state in response to the presence in a 
corresponding cell of one of said minutiae and to a 
second state in response to the absence in the cor- 
responding cell of any of said minutiae. 

11. A non-invasive method for identifying medical pa- 
tients, comprising the steps of 

(a) identifying minutiae on the body of a known 
patient, said minutiae corresponding with 
branch points of blood vessels; 

(b) storing a collection of minutiae data charac- 
teristics of the minutiae for the known patient in 
a memory to define a reference collection, said 
reference collection being unique to the known 
patient; 

(c) sampling the minutiae data of an unknown 
patient in a selected location of the unknown 
patient's body to define a sample collection; 

(d) comparing said sample collection with said 
reference collection, whereby the identity of the 
unknown patient can be confirmed when said 
sample and reference collections correspond. 



12. A method as defined in claim 11, and further com- 
prising the steps of 

(e) detecting any changes in said sample pat- 
^ t ern with res pect to said reference collection; 

and 

(f) analyzing the changes to diagnose the oc- 
currence of a medical event in the patient. 

10 

13. A non-invasive method for diagnosing a medical 
condition in, an individual, comprising the steps of 

(a) establishing a reference collection of series 
'5 of time-varying minutiae data corresponding 

with a plurality of known medical conditions, re- 
spectively, said minutiae data being derived 
from characteristics of minutiae associated with 
branch points of blood vessels in a human be- 
20 ing; 

(b) generating time-varying minutiae data from 
the individual having an unknown medical con- 
dition; and 

25 

(c) correlating said generated minutiae data 
with said reference minutiae data to diagnose 
the medical condition when a match between 
said generated and reference data is obtained. 

JO 

14. Apparatus for identifying the use of a substance 
such as alcohol and drugs by an individual compris- 
ing 

3S (a) means for identifying minutiae data in Ihe 

face of the individual which are responsive to 
the ingestion of a substance by the individual, 
said minutiae data being derived from charac- 
teristics of minutiae associated with branch 

^0 points of blood vessels of the individual; 

(b) minutiae data generator means for generat- 
ing a first collection of minutiae data of a sub- 
stance-free individual and a plurality of second 
collections of minutiae data of an individual af- 
ter the ingestion of a plurality of different known 
substances; 

(c) means for storing said first and second col- 
50 lections of minutiae data in a database: and 

(d) means for comparing a subsequent collec- 
tion of minutiae data of an individual with said 
first and second collections to determine 

55 whether the individual is substance free and to 

determine what substance the individual has in- 
gested where the Individual is not substance- 
free. 
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15. Apparatus as defined in claim 14, wherein said mi- 
nutiae data identifying means comprise 

(1) an imaging device for producing a first sig- 
nal representative of sensed characteristics of 5 
the individual; and 

(2) a minutiae generator connected with said 
imaging device for receiving said first signal 
and for producing in response thereto a second 
signal representative of minutiae of the individ- 
ual, said second signal being delivered to said 
minutiae data generator means. 

16. Apparatus as defined in claim 15, wherein said im- '5 
aging device comprises an infrared camera for 
scanning an individual over a period of time, said 
camera responding to thermal changes in said mi- 
nutiae resulting from the ingestion of a substance, 
whereby said first and second collections of minu- 20 
tiae data comprise thermal signatures of sub- 
stance-free and substance-affected individuals, re- 
spectively 

17. Apparatus as defined in claim 16, and further com- 25 
prising processor means connected with said minu- 
tiae data generating means for generating time-var- 
ying thermal signatures from the scanning of the in- 
dividual by said infrared camera. 

30 

18. Apparatus as defined in claim 17, wherein said 
processor includes correlation means for determin- 
ing the closest match between a subsequent collec- 
tion of minutiae data and one of said first and sec- 
ond collections of minutiae data. 3S 

1 9. A method for recognizing whether an individual has 
ingested a substance such as drugs and alcohol, 
comprising the steps of 

40 

(a) identifying minutiae data in the face of the 
individual which are responsive to the ingestion 
of a substance by the individual, said minutiae 
data being derived from characteristics of mi- 
nutiae associated with branch points of blood -^s 
vessels of the individual; 

(b) generating a first collection of minutiae data 
of a substance-free individual and a plurality of 
second collections of minutiae of an individual so 
after ingestion of a plurality of different known 
substances; 

(c) generating a subsequent collection of minu- 
tiae data of the individual; and 55 

(d) comparing said subsequent collection with 
said first collection to determine whether the in- 



dividual is substance-free. 

20. A method as defined in claim 19. and further com- 
prising the steps of comparing said subsequent col- 
lection of mi nutiae data with said plurality of second 
collections of minutiae data in the absence of a 
match with said first collection of minutiae data in 
order to identify the substance that the individual 
has ingested. 

21. A method as defined in claim 20, and further com- 
prising the step of correlating said subsequent col- 
lection of minutiae data with said first and second 
collections of minutiae data in order to determine 
the closest match therebetween. 

22. Apparatus for determining what portion of a random 
population has ingested a substance such as drugs 
and alcohol, comprising 

(a) means for generating and storing reference 
collections of minutiae data corresponding with 
a substance-free individual and with sub- 
stance-affected individuals, said minutiae data 
being derived from characteristics of minutiae 
associated with branch points of blood vessels 
on an individual; 

(b) means for generating current collections of 
minutiae data from individuals in the random 
population; and 

(c) means for comparing said current collec- 
tions of minutiae data with said reference col- 
lections of minutiae data in order to determine 
the portion of the random population that is sub- 
stance-free and the portion of the random pop- 
ulation that is substance-affected. 

23. Apparatus as defined in claim 22, wherein said ref- 
erence collection generating means generates dif- 
ferent collections of minutiae data in accordance 
with different known ingested substances, whereby 
the portion of the random population which has in- 
gested each known substance can be identified. 

24. Apparatus as defined in claim 23, wherein said cur- 
rent collection of minutiae data generating means 
comprises 

(1 ) ) an infrared camera for scanning and track- 
ing individuals of the random population, said 
camera producing a first signal representative 
of a thermal signature for each individual, and 

(2) a minutiae data generator connected with 
said camera for receiving said first signals and 
for producing a collection of minutiae data cor- 
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responding therewith, respectively, whereby a 
plurality of collections are generated, one for 
each individual in the random population. 

25. A method for determining what portion of a random 
population has ingested a substance such as drugs 
and alcohol, comprising the steps of 

(a) generating a plurality reference collections 
of minutiae data corresponding with substance- 
free and substance-affected individuals ; 

(b) generating current collections of minutiae 
data for individuals in the random population, 
respectively; and 

(c) comparing said current collections of minu- 
tiae data with said reference collections of mi- 
nutiae data in order to determine the portion of 
the random population that is substance-free 
and the portion of the random population that 
is substance-affected. 

26. A method as defined in claim 25, wherein said ref- 
erence collection of minutiae data include different 
patterns corresponding with known ingested sub- 
stances, whereby the portion of the random popu- 
lation which has ingested each known substance 
can be identified. 

27. A method for annotating an image of the human 
body comprising the steps of , 

(a) generating an image of the human body with 
an infrared camera; and 

(b) superimposing on said image the location 
of minutiae detected by said camera, said mi- 
nutiae corresponding to specific branch points 
of blood vessels of the body 

28. A method as defined in claim 27, and further com- 
prising the step of identifying particular minutiae to 
serve as reference points. 

29. A method as defined in claim 27, and further com- 
prising the steps of generating a medical image of 
the human body and annotating the medical image 
with said minutiae. 

30. A method as defined in claim 29, and further com- 
prising the step of identifying particular minutiae to 
serve as reference points. 

31. Apparatus for annotating an image of the human 
body comprising 

(a) means for generating an infrared image of 



the body; 

(b) means for generating a pattern of minutiae 
from said infrared image, said minutiae corre- 

5 sponding with s pecific branch points of blood 

vessels of the body; 

(c) means for generating a medical image of the 
body; and 

10 

(d) means for annotating said pattern of minu- 
tiae on to said medical image. 

32. A method for maintaining the position of a surgical 
'5 instrument relative to a surgical site during a surgi- 
cal procedure, comprising the steps of 

(a) identifying minutiae in the vicinity of a sur- 
gical site on a patient, said minutiae corre- 

20 sponding with specific branch points of blood 

vessels of the patient; 

(b) generating a reference minutiae pattern for 
a stationary patient; 

25 

(c) generating a reference position of a surgical 
instrument with respect to said reference pat- 
tern; 

30 (d) detecting deviations from said reference 

pattern owing to movement of the patient and 
from said instrument reference position owing 
to displacement ofthe instrument; and 

3S (e) repositioning the instrument with respect to 

said reference pattern to accurately position 
the instrument at the surgical site. 

33. Apparatus for maintaining the position of a surgical 
^0 instrument relative to a surgical site during a surgi- 
cal procedure, comprising 

(a) means for identifying minutiae in the vicinity 
of a surgical site on a patient, said minutiae cor- 

^5 responding with specific branch points of blood 

vessels of the patient; 

(b) means for generating a reference minutiae 
pattern for a stationary patient; 

50 

(c) means for generating a reference position 
of a surgical instrument with respect to said ref- 
erence pattern; 

55 (d) means for detecting deviations from said 

reference pattern owing to movement of the pa- 
tient and from said instrument reference posi- 
tion owing to displacement of the instrument; 
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and 

(e) means for repositioning the instrument with 
respect to said reference pattern to accurately 
position the instrument at the surgical site. 5 
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